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ABSTRACT 

As a supplement to the principal reports, the 
underlying management system of the U. S. Naval Academy Self-Paced 
Physics Course is presented in this documentation report. The 
management system is discussed on three "levels" which are intrinsic 
in the programmed-instruction format, focused on remedial activities, 
and related to the handling of student performance data. Detailed 
information is given in connection with the recommended course 
implementation procedures, nature and form of tests, methods of 
scoring and recording scores, kinds of feedback provided by students, 
methods of presenting feedback, record-keeping procedures and forms, 
and computer-managed operations. Also described are the methods of 
utilizing data generated during the course to evaluate and improve 
the course material and presentation strategies. Besides a flow chart 
reflecting the responsibilities of students, clerks, instructors, and 
managers, a total of 25 exhibits and illustrations are provided. 
(Related documents are SE 016 065 - SE 016 088 and ED 062 123 - ED 
062 1250 (CO) 
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1. INTRODUCTION 
Background 

This Report is the product of an extensive educational 
research and development. effort that was conducted in conjunction with 
the U.S. Naval Academy and the N.Y.I.T. Learning, Management & Resources 
Center. Documented in this manual are procedures which are based on the 
educational philosophy and management techniques developed during an O.E. 
sponsored, three-year experiment. We do not purport to have created the 
"perfect" management system, but rather a system which contains the found 
tion for practical implementation and the ability to evolve when dynamic 
educational needs become apparent. 

The Management System operates on three "levels." TLe course 
in essence, is made up of sophisticated, multimedia, multi-resources, 
'independent study materials presented in a branching programmed-instruc- 
tion fomiat, with a maximum of response demand frequency. The student's 
progress through a variety of learning experiences, ranging from tests 
and conventional lectures to single concept audiovisual materials and 
tutorial sessions, is guided by his own responses and is student self- 
iDanaged to a high degree. 

The first level of management is, therefore, instrinsic in 
the programmed-ins true tion format. 

The second level of management derives from instructional 
activity focused upon tutorial remediation as indicated necessary by 
immediate analysis of student performance on the study guide response 



sheets and progress (criterion^referenced) checks. This remediation may 
be individual, or grouped by student section, and may be based, in part, 
upon analysis of individual student performance alone, or as related to 
parallel performance of the rest of the sections or to the class as a 
whole. The tools for this management are the students* responses and the 
item analysis performed by the clerical data management support groups. 

The third level is provided by the output of the computer or 
Data Management Group ^which utilizes the performance data from the pro- 
gress checks and quarterly diagnostic exams to* provide a more in-depth 
prescription for independent aud group remediation. 

The input of all student responses into the computer also 
provides the basis for the revision guidelines for use in optimizing the 
course materials. 

One of the many factors determined during the experiment was 
that, using the system^s approach, the data handling capability of the 
computer in conjunction with the exercise of the instructor's managerial 
judgment represented the best of two distinct management methods. 

The procedures set forth in this manual reflect the most cur- 
rent concepts in course management deriving from the experiences of this 
contract. 

Purpose 

The procedures for implementation of the management system 
are described in two sub-manuals.: 

1) The Course: & Data Management Manual, and 

2) The Performance Data Analysis for Course Revision 
& Optimization. 
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The Course & Data Management Manual describes to the course 
manager: a) requirements for planning and preparation needed prior to 
course implementation; and b) procedures, decisions and record keeping 
required while the course is in progress. 

The Performance Data Analysis for Course Revision & Optimiza- 
tion contains descriptions for utilizing the data generated during the 
course in order to evaluate and improve the course material, material 
presentation and strategies, and the management system itself. 
Instruments 

In order to accomplish the several levels of mauagement , a 
large number of recording methodologies, reporting devices, test forms 
and instructions to the students, data manager, instructors and course 
managers have been created. Examples of these instructions have been 
incorporated herein, directly or as exhibits. 
Associated Documents 

The following documents are a necessary part of understanding 
and implementing the full computer managed portion of the system: 
Course & Data Management Manual 
User's Manual: A350 
Operator's Guide: A380 
Program Logic Manual: A370 

Systems Manual: A360 • 
Performance Data Analysis for Course Revision & . Oprim'ization' 
Systems Descroption 
User's Description 
Supplement 
Programs 



Additionally, many pertinent technical reports have been sub- 
mitted during the course of, and as final reports to, the multimedia 
physics development program. These reports contain the rationales and 
experimental data analyses upon which the management system has been 
built. In these technical reports will be found the justification for 
all the elements of the management system. 
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2. COURSE AND DATA MANAGEMENT MANUAL 

2.1 Learning Resources t ^ Mnn i.^od^ 

Several purpose i. . the program's development 

and operation. The Academy iw i^iuvided with a cost-effective physics 
course designed with the most modern educational technology. The experi- 
ence garnered in the construction of the program is incorporated into a 
course development model, to serve as a prototype for construction of 
similar programs in other hard science courses. Extensive record keeping 
capabilities of tlie program allow it to serve as a vehicle for educa- 
tional research. 

The course is individualized, self-paced, and self-healing. 
By offering media options and optional routes through the learning mater- 
ials, each student's learning experience is individualized. Likewise, 
within broad time constraints, the student can progress through these 
materials at his own pace. 

The self-healing aspect of the program pertains to learning 
'materials and processes. These are improved by ah iterative process of 
successive tryouts and revisions. In this way, the program undergoes an 
empirical optimization procedure. 

Most standard topics in introductory classical physics aro 
included in the two-semester course: mechanics, wave phenomena, elec- 
trictty, magnetism, and optics. One unusual feature Is the omission 
ot heat 'and thermodynamics in favor of more intensive developments in 
mechanics and optics to suit the Academy's particular needs. The sub- 
ject matter is loosely defined as university phyaics with calculus at 
the level of Halliday and Resnick. 



optimization of: the program must be attained empirically, 
since no satisfactory predictive theory of educational psychology is 
knovn. This situation is well suited to a systems approach when the out- 
put of the system can be fed back to modify the system input. 

This approach requires that the course objectives are clearly 
defined and measurable so that the output of the system is quantitative. 
Consequently, over a thousand measurable behavioral objectives (MBOs) 
were developed to completely specify the performance which the program 
should elicit. MBOs fall into two categories:' terminal objectives (TOs) 
which describe the desired final student behavior, and the subordinate 
enabling objectives (EOs) which are steps toward the terminal behavior. 

The TOs constitute a complete description of course content 
and are represented in the course by central core problems. When a stu- 
dent can answer a core problem correctly, he is said to have achieved 
that TO. In this way a student knows exactly what is expected hir^ as 
to content and level of proficiency. 

When a stud«Bt cannot answer a core problem after a single 
exposure, he can execuai^«»bordinate enabling problems which correspond 
to the EOs. At the endsaigz;^ enabling sequence, the student is presented 
with another version of -lifae - core problem to check his achievement of <:he 
TO. All these problems ,:aEej contained in the Problem and Solution Book 
volmes. 

^ ' ' Videotape pacBSentations are available for forty-nine topics. 
These tapes average about fifteen minutes apiece. Illustrated texts and 
talking books (taped voice-over illustrations in book form) are available 
with essentially the same information content as the videotapes. 
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(Computer-assisted instruction (CAI) was initially included as a paralleil 
path for topics in mechanics, primarily to be compared with the other 
options for cost effectiveness, and was deleted as a learning option in 
the final revision,) Conventional physics texts aire also included among 
the available Icniuxu, it \irials. 

Each sLudtiiL is provided with a Study Guide which directs 
him through the problem books and various media assignments. When the 
student completes a specified assignment (roughly approximated by a 
chapter in Halliddy and Resnick) , he schedules* a Progress Check (test) 
on the material. This criterion check does not influence his grade but 
is used for management and remediation purposes. Grades are determined 
by quarterly tests and final examinations for each semester. 

Minor remediation is accomplished by distributing a remedial 
sheet assojc^ted with each iHawblem missed on the l^rogress Check. These 
sheets have^a statement of tfe^r appropriate core ^^moblem together with 
ref erences:;iCD pertinent auxflaary material. Moi3E2serious remediation 
is providedSyy individual tuJsaacial sessions with professor. 

ZThe laboratories have as their objectives the measurement of 
fundamental irphysical quantities , including the processing and recording 
of this daitaswith an error analysis. An innovative aspect of the dats 
analysis isEiHiiat a dialogue may be established between the student and 
t>ie: compiu«!Li whi ch would culminate in the student^s achievement of the m 
objecti;v^f, "Ihifa computer-dialogue lab format is not an essential ele- 
ment of: the-:attil timed ia learning materials , and in the current imple- 
mentation tte^ Navy physics sl:aff is using a conventional laboratory 
experience :£dt the self-paced, course, , 



2.1.1 Problem and Solution Ho:ok (P & S) 



This is bound study material which contains the work of 
forty-five (45) segments for a semester. Each Problem and Solution Book 
contains: 

a) A problem section In which the questions are 
presented in a sequential order; 

b) A solution section in which the correct method 
of answerins is presented; 

c) tnfonaation Panels strategically placed through- 
out the problem section. 

A descrii)tion of the topics covered by the segments 



is as follows : 

Measarement ard Vecitorai 

Vector MultipliCcctlflacE^ffiBi Velocity 

Motion in Oae and Two .B&Eens ions 

Newitrn * s Laws o f Mo t xtan: 

lAHTrrr Motion: Uraction: 

VmBcsrcn Ciicular Motion 

Worlc and Energ}' 

Conservatiun of Energy 

Matlon of the Center o£ Mass 

lEinear Momentum 

Impulse and Collisions in One Dimension 
CoU-isions iu Two Dimensions 
Gravitation 

Gravi t at ional Bdtent iaUEnergy 
Electric Charges and COTtlomb ' s Law 
Hi© Electric FiaeOid 
^ectric Field ^oblaaBs 
laNptric Dipoles and Hlfectric Flux 



<^au s s*s Law 

fffeestric Potential 

El metric Potemtial Energy 

G^: ^iicitance 

f gy Stored in Capacitors 
C>: "nt and Resistance 
F metrical Energy & Electromoinroe 
Force 

Circuits & Kirchhoff 's Rules 
Ammeters & Voltmeters 
Charge in a Magnetic Field 
Current is a Magnetic Field 
Magnetic lELux & the Earth's 

Magitetic Field 
Ampere's Law 
The Biot-Savaad: Law 
fiaraday's Law of Induction 
Self Inductance 
The RC Circuit 
The LR Circuit 



Ifcenty additional segments wtre-. prepared by the Naval Academy staff. 
2.1.2 Remedial Sheets 

The student receives a remedial sheet for each question 
he misses on the Progress Check (PC). The remedial sheet contains: 
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a) A restatement of the appropr±ate core 
problem; 

b) Reading assignments, and 

c) Related problems when appropriate. 

The remedial sheet serves as a muLtl-purpose remediation 
eleit«liri3fepending upon the student's achievement level in specifically 
identE^M: problems on tbr-i Progress Check. For example: 



1) 
2) 

3) 



faasndent Achievement 
i33ErEBrogress Check is: 

:WD — 100% correct 

^ — 70% correct 



c»ir ISbss correct 



Student Use of Remedial 

Retains for revieBTr purposes 

Reviews isemedial asni schedules 
Progress Hheck retafee only on 
appropriafia core pral ilems 

Reviews :32snediaL,.-3a3Bedules 

t U t or r^l^aS S i nn -.^rrP 'l^.'r ngT-e-g Ch e ck 

retaka. 

It is necessary, therefiaasE,. that-3L£:sufSSicient number of 
iee .-iteswwBwL ill advance for each Process Check itHM to be used, 
-of remedials follow. 
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SAMPLE REMEDIAL SHEET 



SEGMENT 1 
MEASUREMENT AND VECTORS 

Problem 6: Addition of Vectors 

6.- A plane travels 'AO miles due north, then changes its """^ J^J^ 
direction of 37°east of north and travels for 50 miles, finally it 
travels for 30 miles due east. Its total displacemevt is: 

A. 100 miles 

B. 120 miles 

C. 100 miles at 37° east of north 

D. 120 miles at 37°' east of north 



Reading Assignmenn 
Halliday and Resnick: Ch. Z^-^^mst. 1-3 

Semat and Blumenthal: Vol. ^L^. 2. Er 11-18. 36-40 



Schaum: 



Related Problemsr 

Ch. l,:llDS. 1. 2, 14 
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SAMPLE REMEDIAL SHEET 
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SEGMENT 1 



MEASUREMENT AND VECTORS 



Problem 10: Resolution of Vectors 




Find the com,:DiiaiiEts ?,^aaxdi^ ofcrtfae vector R, whene^^ector 31 is the 
resultant (uum) of the:inEc:tors,iA,iB:, and C. Use thestxcoordiwate system 
indicated. 



A. 




sine Ry 


:c ase 


B. 




cose Ry 


===A. sine 


C. 


^ =2 


- A sine 


.Ry = C - A cos9 


D. 


B^ = C: 


- B cose 


- A - B sine 



HaaiSday and Re snick: 
Senatft and Blumenthal: 

Schsom: 



Reading Assignment : 

' Ch- 2, Sect, 35 

¥ol. I, Ch. 2, :Er 3M6 
Related IPrrnblems : 

Ch. 1, Nos. 10, .11, A 
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2.1.3 Audlovisuals 

The Audlovisuals, representing the various media support 
materials prepared for this course, consist of one complete set of vidtD- 
tapes, talking books and illustrated bonlcf?, All ilni ui the media have 
bBsically the same information content and cover the topics sho\m on the 
list below. 

The criiteriazaised to determine when an Audiovisual would 
be prepared for a particmlar topic was as follows: 

a) If t:he topic concerned the concept of motion, or 

b) If tiEhe sui3ject matter was rated difficult and the 
text T?as not sufficiently elaborate, or 

c) If xDEcterial for enrichment did not appear in the 
text at :^1. 

All of t&e Audlovisuals were prepared in order to pro- 
vide nKiltimedia paths for the student to optimize and individualize his 
learning process. 

The videotapes contain -prerecorded illustrated lectures 
to be jSisplayed on individual monitors. The average playing time is 
:al)out SLfteen minutes. 

The talking book consists of a tape recorded lecture 
(von^rther fspen reel or cassette) and an accompanying bookxet containing 
the illustE^tions for that lecture. . • ■ 

The ilLiEs:trated book contains the same lecture material 
3s the traifeing book but ias presented without the audio medium. n-., 

A i?^fe^ttz^pe, tailiEing book and illustrated book were 
pKtmreaS fox each of tibee lollowin^i topics: 



Sequence No* Title 

1 Projectile Motion 

2 N:^wton*s First Law 

3 Newton's Second Law 

4 Newton's Third ,Law 

5 Atwood's Machine 

6 Circulaar :Motion 

7 Work Whcaa Force Varies in Magnitude 

aad Direction 

8 Kinetic Energy 

9 Potential Energy 

10 Conservartion of Energy 

11 Movememt of Center of Mass 

12 Conservation of Momentum 

13 Impulse and Momentum 

14 Collisions 

15 Gravitation 

16 Angular Quantities Treated as Vectors 

17 Moment of Inertia 

18 Rolling Without Slipping 

19 Angular Momentum 

20 The Gyroscope 

21 Simple Harmonic Motion 

22 Energy in SHM 

23 Pendulums 

24 The Reference Circle 

25 Traveling Waves 

26 TFave Velocity 

27" Combination of Waves 

28 Standing Wawes 

29 :Beats and t3ie Doppler Effect 

30 Reflection and Refraction 

31 Huygens' Pr±nciple 

32i Total Internal Reflection 

33' Images by Reflection 

• . 34 Images by Thin Lenses 

35 Calculation of E for an Infinite 

Uniformly Charged Wire 

36 Deflection of EIsectrcBDS in an 

Electric EiielE 

•.37 Flux 

38 Calcalatlon of lE Using Gauss's Law 

39 Gapaaciitors 

40 The Capacitor im Actiflaa 

41 Kirchhoff's Rules 

42 Definition of a *B" meld 

43 ^Foxce Between PssHsallel. Curxent- 

CarryiDg iGondnc t ors \ 

44 .Ampere's Law As^Kled to a Long \ 

S tr aigflat: Con#ac tor ' \ 

45 The Law of BiofeSSavart * 

46 Taraday's Law Induct iom 
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47 Motioa of an Electron in Combined E 

and B' Fields 

48 LR Transients 

49 Transients 

2.1.4 Texts 

. The principal text which is issued to each .s«B»aent is 

Halliday & Resnick; Physics for Students cf Engineering aiad.3c:;i3encc « 

Supplementary reading assignments are to be found in oUiei: tSKts. vhich 

are made available during the classroom period . These tex±s ^a^DEr 

Sears and Zemansky, University Physicr 

Albert Baez, The New College Physics — A Spfeal 
Approach 

Shortley and Williams, Elements of P hysics; 

2.1.5 Instructor 

One of the major differences between thxs:.:jG3mTse and a 
conventionally taught course is thpt role of the ins true to-r.. "^lie-ins true- 
tor is available during class sessions but does not give- a Jtecituz^e. His 
primary function is to assist students with their problemsi-im-a one-to- 
one basis or to schedule additional assistance in a tu toacialliii M -t^x tra- 
instruction session. Ttd^ "individualized** instrucirLon a^aSEfcitD iden- 
tified weaknesses or needs constitutes one of the basic foa itiffliHl imms upon 
which the course has been structured. 

An average instructor/stiudent ratio of l±20rwas: 
scheduled but varied somewhat in practice because of studieaxt: .dro?p&:outs.. 

Further description cif tiie instructor's xoie isr_t:o be 
found in this report under Section 2.3.2. 1^ entitled "Notes to thaejlnstru 
tor of Self-Paced Physics." 



Certain "demonstration" type lectures were prepared 
because of the appropriateness of the subject matter. For the first 
semester these demonstrations covered the following topics: 

Linear Momentum 

Angular Momentum 

Wave Motion 

Sound 

Geometrical Optics 

The second semester covered these topics: 

Electrostatics 

Electromagnet ism 

Physical Optics 

Polarization 
2.1.6 Laboratory Sessions 

Two alternativp seif-paced laboratory approaches were 
■developed in conjunction with the physics course. 

The conventional approach involved twenty (20) experi- 
;mental set-ups in. the laboratory at any one time. Duration of the experi- 
ments was planned to range between 1-3 hours; however, the experiment 
need not be completed in one session. Ideally, no more than two students 
.a± a time are recommended to participate in an experiment. A cbmplete 
:dBscr%tion of the laboratory procedure and policy is in Section.-2v3.2. 4 
a£ thfe /.report , including an identification of the twelve (12) eKptvirr.. 
mesnts scheduled for the first semester. This conventional lab procedure 
was desreioped by the Naval Academy physics staff. 
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The laboratory program, in conjunction with the multi- 
media .physics course developed by New York Institute of Technology, con- 
sists of fifteen (15) independent study, computer-supported experiments, 
covering the following topics: 

1. Probability and Exper imental Errors in Science 

2. Permutation and Combination 

3. The Direct Measurement of Length 

4. Electric Fields 

5. Current Balance 

6. RC Transients ^ . 

7. Projectile Motion 

8. Graphical Analysis of Data and the Method of 
Least Squares 

9. Collisions 

10. Conversion of Linear Momentum 

11. Potential 

12. The Magnetic Circuit: and Hysteresis 

13. Measurement of Frequency . Response 

14. The Inclined Pl^ne 

15. Error Analysis 

Plus Overview and Summary Report, Instructor's Guide- 
lines and Answer Lists. 

This program is described in detail in the uep'^rate sub- 
mission entitled "The Multimedia Labs." 
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2.2 Management Devices and Controls 

The management tools that were designed to operate with the 
course material while the course is in progress consist of the Study 
Guide, the Progress Check and the Quarterly Diagnostic. This section 
more fully describes these items and includes sample reports. 

2.2.1 Study Guides, Progress Checks (including remedial 
progress checks) and Quarterly Diagnostics 
Study Guide 

This guide is used in conjunction with the Problem and 
Solution Book. Its purpose is to: 

a) guide the student through the course segments; 

b) provide the student with a mechanism for testing 
his understanding of the subject matter with immediate 
feedback; 

c) indicate reading and audiovisual assignmenLo, and, 

d) provide paths for reinforcement and/or remediation. 
An intrinsic element. of the self-paced concept, embodied 

in the Study Guide, is thq mechanism for the student to proceed through 
the course as fast as. he is able to grasp a thorough understanding of the 
subject matter. 

1' It is recommended that if a student schedules a tutorial, 

he 'bring with him his completed Study Guides as an additional aid for the 
Instructor to determine the student *s problem areas. 

An example of a Study Guide page is included in this, 
section but a full description of its use is to be found in the Student 
Manual, Section 2.3.2 of this report, Instructor/Student Orientation. 

19 
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Profiross Chocks (including Remedial Progress Checks) 

The Progress Check is a management device designed to 
aid the student as well as the instructor and course manager. 

It indicates to the student to what degree he has 
achieved a mastery of the Terminal Objectives; it represents the first 
formal feedback to the instructor in order to evaluate individual stu- 
dent success with the course an early stage so that he can take cor- 
rective action; and it forms the basis for the course manager to judge 
overall student progress in the course and ascertain problem areas. 

When a student completes a certain nu iber of segments 
he schedules a Progress Check with his instructor on the material covered. 
(A full description of the Progress Check procedure is covered in Sec- 
tion 2.4.2.1.2 of this report.) 

For the initial half of the fall semester of 1970, 
Piugr^ss Checks were administered on groupings of segments as follows: 

a) Segments 1, 2, 3, 4, 5 

b) Segments 6, 7, 8, 9, 10 

c) Segments 11, 12, 13, 14 

The Progress Check is an "individualized" test in that 
it consists of ten qtiestions randomly selected from a large bank of test 
items but is not used for the student's grade. (Refer to Preparation of 
Progress Checks below.) 

The satisfactoty completion of the Progress Check per- 
mits the student to continue with the. next series of segments. 

A student answering correctly 70% or less is given a 
remedial Progress Check on the particular core questions missed on his 
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first Progress Chock; i.e., the remedial Progress Check contains vari- 
ations of each item incorrectly answered. The remedial Progress Check 
is usually scheduled by the student after he has completed appropriate 
remediation. 

If the student answers correctly on 40% or less on the 
Progress Checks, he schedules a tutorial session with his instructor 
prior to taking the remedial Progress Check. 

Progress Check results are summarized by the Data Man- 
agement Group and submitted to the course manager for review. 

The first page of a sample Progress Chock is submitted 

in this section. 

Preparation of Progress Checks 
. The preparation of Progress Checks, while done manually, 
is designed to be compatible with a CMI system. That is, the questions 
were, coded with unique numerical i^^ intification and were, for the most 
part, multiple-choice to facilitate compu^ei verification of the manual 
grading. 

The content of each question was determined by refer- 
ence to the core questions in the Problem and Solution Book and to the 
Terminal Objectives which the core questions support. 

Questions used were of the multiple-choicr , true- false, 
or completion type. 

Five variai^Vi ^h question were produced,:. usually 

by rearranging numerical parameters within die text of the qufes^titfr rather 
than restating the question in completely different language. Although 
the Progress Checks were not used for midshipmen's grades, and one 
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Progress Check could have been given to all midshipmen, additional copies 
were required for midshipmen who scored low enough to require retakes. 

The original questions were stored on cards, one card 
per question. The first Progress Check on each group of segments was 
limited to ten questions. Tests were prepared by putting several cards 
in a holder, masking off certain administrative information, and producing 
a ditto master on a commercial copier. The ditto master Lhon was used to 
produce the number of copies required. Clerical personnel selected, from 
the appropriate group of questions and versions, the specific cards that 
made up a test. A pattern was set up to ensure use of all versions of 
all questions. Questions required for retakes depended on v^hich ques- 
tions the midshipman missed on this Progress Check. Requests for those 
questions were passed to the clerical personnel in the form of graded 
Progress Checks or listings of specific que&tion numbers. 

Course policy required 48 hours notice of intent to take 
Progress Checks, but this lead time appeais generous for the production 
system employed. 

Time alloted for midshipmen to complete a Progress Check 
varied with instructor, but was usually -50 minutes. Many midshipmen 
frequently took longer. Upon completion of the Progress Check, the 
instructor, having been provided an answer key, graded the Progress 
^Check and either returned it to the midshipman for review or i^«€§d it to 
clerical personnel for statistical analysis and preparation ol reifeeike-^'ro- 
gress Checks. 

Vijartciiy Diagnostics 

At four scheduled class periods during the semester, a 
Quarterly Diagnostic is administered. Each studeAt section receives an 
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individua' t. These Diagnostics are used both for grading pur- 
poses am 6ii3 :St further the instructor in diagnosing any weak areas 
of each st • achievement. The items thoroughly cover the Terminal 
Objectives during the period. Generally, these test items are 

created to student's performance on general and specific con- 

cepts of pb^ysfr romplex mathematical procedures are not emphasized. 

!JMe diagnostic problem types are categorized as follows 
^jLagnostic Problem Types 

~R Recognition of material that should have 
been memorized. 

m Conceptual understanding; single concepts 
isolated from problem applications. 

,iR Conceptual relationships; recognition and/or 
selection for use in problems but no actual 
solution required. 

« 

At present, all Diagnostic items are multiple-choice 
questions car: pm^ 3ms--to facilitate grading. At the next class period, 
each studexnt ^rves a personalized "Student T.O. Key Sheet" which indl 
•cates his .^.*crf for the test. Also, the test items missed are circled, 
and he is #ixttted to the particular locations within each segment he 
should review. 

(A full description of the Diagnostic procedure is 
Included in Section 2.4.2.1.3 of this report.) 

; The results of the Diagnostic form the basis for a 

coiaplefce.' evaluation of the student and the course effectiveness. Sample 
summary reports are included in the following section; a complete set 
of report f^h: to be found in Section 2.4.2.1.3. 
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SAMPLE STUDY GUIDE 
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SAMPLE PROGRESS C ECK (fl^-— p^,g^; ) 



Nsme 



ID 



Section 



Date 



Prrsi^sr^^^ ck Score 



out of :io 



Progrw^ Check fl) Seg H-^S 
Reizzz ^^o. Made 



1-1. 1 The c:imEE5 of theiSgaire shown w^sire measuxed vmih different 

faa^fernssn fat The arrEa-:should be cxsiiressed in hownnany signifiiGsnt 



A. I 

B- L 

C. 4^ 

D. 66 



2, 0 in 



-4 .M) in 
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internal Its rate can be?^: expressed in 

A. imi-sec 

B . s^ec jQEsr ft 
C . mi p^: hr 

::d . fcr/ft ""^ 
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2.2.2 Sample ^lanagemcnt Report^: : 

This section contains saropli? nrini^eincnt reports whicii 
reflect actual data* A complete set of mHEsg*.' iritrnfe xeportt forms are 
included in Section 2.4*2. 

The reports shown here srse: 

a) Progress Check S 

b) Diagnostics 

(1) All Sections--SteEr» Dajs crxhutixni 

(2) TO Smiinary Compairi-^^: ^'- --^ercents 

(3) TO Summary Compar ' ^3Ti — Students 

(4) Bar Chart PercenHS^ Mstributioxis 
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2.3 Control Points and Orientation 

Incorporated in this section is a pictorial description (flow 
chart) which reflects a general overview of the operational functions of 
the course. Various control points and lines of communication/responsi- 
bility have been identified in the flovj chart for the benefit of the 
instructor and course manager. 

Ijescriptions which amplify these items are in the Instructor/ 
Student Orientation section and other references found throughout this 
report. 
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2.3.2 Instructor/Student Ortentatton 

This section contains information which should be re- 
viewed by the instructor to familiarize himself with the course. The 
descriptions are divided into five parts consisting of: 

a) Notes to the Instructor 

b) The Management Sequence 

(The following three items consist of 
documentation which is distributed to 
the students during the orientation 
session.) 

c) Course Policy 

d) The Self-Paced Laboratory 

e) The Student Manual 

2.3.2.1 Notes to the Instructor of Self-Paced Physics 
1 . Introduction 

The methods and operation of the self-paced physics 
course may seem strange to new instructors as well as to the students 
. This information is presented to assist the instructor in developing ^lis 
individual class policies. It is presumed you are familiar with the 
Student Manual and Course Policy Statement. 

'2. Objective 

The objective of the course is to enable each midship- 
man, to complete the tasks defined by the Terminal Objectives (TOs) . If 
you /have not dnne so previously, you should read the TOs, as they consti-- 
jtute; the most accurate definition of course content. Because of the way 
the Problem and Solution books have been constructed, successful comple- 
tion of all the core questions should cover all the TOs. Since the core 
questions were also designed to provide a path for fast students, they 
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are frequently complex problems that combine elements of several TOs. 
Due to the limited time available for testing, the body of TOs is sam- 
pled randomly during Progress Checks and Diagnostic Tests, 

3 • Class Atmosphere 

There are few constraints on how you use class time 
to move the students through the material. If your class size permits, 
you are encouraged to usr, RooiTi 203 as your regular classroom. Initially, 
a certain amount of encouragement may be needed to steer the midshipmen 
to the various media. You should try as many of the media yourself as 
time permits so you can recommend a particular Audiovisual if a mid- 
shipman is having trouble ±n a specific area. You may wish to add 
additional demonstrations or conduct small topical lectures occasion- 
ally. Comprehensive reviews prior to Diagnostic Tests are frequently 
given* 

4. Student Progress 

One of the by-products of the course organization is 
the early identification of potential failures, before they reach the 
Diagnostic Tests, This early identification can be done most effectively 
by careful screening of Study Guide responses and Progress Check responses 
The individual prescription for assistance is in your hands, but the early 
identification of these individuals and the variety of materials available 
should provide you with considerable flexibility. 

5, Areas of Concern 

a. Minimum Lecture . You, as well as some of 
your midshipmen, may feel uncomfortable, initially, because you are not 
conducting lectures during most of the class time. Experience has shown 
that most students adapt readily to the self-paced class routine within 
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four-iro six weeks. You may choose to lecture frequently; however, you 
will probably have little time left to grade Progress Checks or counsel 
slow students, except in EI (Extra-Instruction) Sessions. Another by- 
product of the course organization is to move a substantial amount of 
student counseling and remedial work into the classroom. 

b. Student Progress . Because of the great amount 
of material covered by the course, you will soon find students dropping 
well behind the average (or, from your view, a desirable) class progress. 
Your success in keeping the class moving will be limited only by your 
imaginaC3.oB. One reason tor the apparently slow class progress may be 
conf usiocL: bcetween a very weak physics student and a good student who 
chooses to **pace" himself to the speed of slower classmates. Careful 
screening, of Study Guide and Progress Check responses can usually separate 
the two. 

2.3/2.2 The Management Sequence 

1^ Each student is issued one prime textbook; 
at least two other supplementary texts are at all times available in the 
physics room or in the library. 

2.. Each student is issued a Student Manual 
intended to supply the student with all the procedural information required 

3. Course work begins with. the issuance of Seg- 
ment 1 of Problem and Solution Book-and the Study Guide for the same seg- - 
ment. The Study Guide is a latent image type on which sequencing infor- 
nation is revealed by means of a special pen. 
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4. The Study Guide features are: 

(a) A reading assignment indicating prime 
reading ands^saaapdementary re-ading, both clearly identified. 

(b) Core problems identified by score lines 
over and under the problem number. 

(c) Remedial loop problems ("enabling pro- 
blems") • • The instructions for short-circuiting, the loops, ol* following 
them, are conitained in the Situdy Guide f^ eaich individual set. 

Cd) Titles .aiiitd directions for Inf occmation- 
Eaaaels contaiiE^-iim the KnafilEra and Solution book.. 

(e) Ti-tifh^s and directions for Audiovisuals. 
These are a\^3l2^Dle in threp. f ormats : 

(1) Videotapes ; 
C2.) Talking books; 
(3^ illustrated texts 

(f) Homework assignment, generally in the 
form of aflMibticmal prohfems ±a the prime text. 

5. Tke Problem and Solution book features are: 
(a) Problems and diagrams in numerical 



sequence. 



each one in a box. 



(b) Core problems identified hy enclosing 



(c) Information Panels preceding care ^:groups . 

(d) Scrambled problem solutions: directions 
for reaching solution is revealed only in the Study Guide when correct 
answer is chosen. 
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(e) Each soItrt±(on for core and core-prime 
questions is followed by a true-false qu€S3±iz?.n whose answer is derivable 
from the solution to which it pertains. These TP's are answered in 
special boxed sections of the Study Gui&i. I^OTE: Each core problem 
which is answered incorrectly requires trfeffit t:2»e student follow the 
remedial or enabling loop which always casnasSianaes with another problem 
having the same conceptual basis as the rxsoos?** ;aBE^ initially missed . 
Such problems are called "core-primes.'^ 

(f) The scasabli^rjEnDce^^ used for the 
solutions is extremely difficult to compntamMisE^ Ife tt±me required to 
sho3ri£-circuit the response pattern is exffflscted "^lie too gireat to make 
it wzerthwhile. 

6. The Progre'ss Chfick. This is a form of test 
which follows a unit of work. The Progress Check is graded by the 
tjacher. The performance of the student is evaluattsed and he is then 
guided into o:ie of the channels indicated baeStow. To be eligible for 
the Progress Check, the student must submt ±:o his instructor all of 
the relevant revealed Study Guides for that uniLt.. 

(a) Using a prefeterrm'TTtpriir -cut -off grade, 
the student is given the "go" signal if his p: pr riKim a nre . is above this 
level. He is also given a set of enrichment suggestions in the form 
of reading, programmed material, -f il-ms , etc. 

(b) If his performance falls below the 
cut-off, he is given a "stop" signa:l with remedials, after which he 
retakes a Progress Check. Questions on these checks will be random- 
ized so that no two students ever take exactly the same examination, 
nor does the same student take the same check on the second round. 
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(c) If his performance falls below cut-off 
on the retake,, he will be given individual t«orial assistance and re- 
quired to take a third test. Disposition of irfe student after the third 
failure will be left to the chairman of the g i lmyjs ics committee at thB 
i^ademy . 

7. Quart^erly DiagnosJt±c Xeists. These tesitis will 
be carefully generalzed to test for recognltifcm snd recall, understauiing 
of concept, ability tt:o recognize concepts TOhtirch appear in probleims,, and 
ability to solvxi: u^xaajJblems . Tkiesse tests will call be ceE the multipiLe- 
choice variety, wxtin a respcm^ rochanisni snnlttable :&r computer gj^ading, 
ifflne of the Quarterly Diagnia©t±cs will replace the miidr-term examination 
and the last of them will be atfeLnistered ab:out one week before the 
standard final examinatirm.. 

8. At the end of each quarter the instructor 
will submit a diagnosis and recommendations based upon Study Guide re-. . • 
sponses, performance on Progress Checks, and Quarterly Diagnostics. Pos- 

'sible recommendations include continuation of::^:equenee> fffaw*tntion cf 
specific segments, further use of other program texts, ardStional tutorials, 
and dropping out • . 
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2.3.2.3 Sample Courrse Policy 



SPZEE SELF-^ACED PHYSICS 
Coumse Policy 



IL, BackfiriOAmid. , This ccyurse is p:art of an expetrlmental education pro- 
giEam -iwhich. rlnas been oon€uctEd by ithe U.S. Naval. Academy and New York 
liustaHrinttc c^f fechnolo^ since summer 1967. The course is character-- 
iizsed luvy ra^atiker unci*i»ni«oxtional teaching methods.^and , as you will see,. 

Each strident: 's proceeding through th^ course at 
his ow»rr - ace. 

Use of instruction v±a texts, prograiraiiad problem 
and soUiu^ion books, videotapes, audiotape record- 
ings, axt^ insutructor consultation. 

2,. Insanructional Methoids . A variety of instructional methods will be 
assailable to you during this course. The large number is provided so 
tiftat you can find xhe combinaitistopr of materials that is most efficient 
iSwt you.. To be sutTe of optimifezaiE^ryour use of classroom time, you arc 
usB^ed tro try each type jaif material at least twice. Th'e materials aval 
dble Ssb : 

Al- Problem and Solution Book. This book., together with the 
St2iidy Guide, is th^ priiiiciLpai material. -Des:cx£ption and Organ- 
±zart±on axe contained im tthe Student Manmail. 

Audiovisuals (JS7*si),. These take several forms; you 
rihould try aOS. , 

(1) Videotapes. Prerecorded lectures on specific 
topics are shown on the large monitor in Rm 203. 
The schedule for this monitor is posted on the 
wall. If you wish to look at a different video- 
tape, you may check roane out and play it on the 
TB^deotzape na chines imiRm 203. 

(2) T<^lking Book (TB!) . This method consists of 
a tape recorded lecture (on either open reel or 
cassette) ani a booklet containing the illustra- 
tions fm: the lecture.. The TB materials are 
checked .But in Rm 203. Cassettes may be checked- 
out ovexnight . 
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(3) Illusitratcd iBook. This iis a short illustraited 
booklet coimtainimg the same lojcturc as the Talking 
®ook, but T^rithouit the audio poirtion. Note that the 
subject maitter of the Video taiPJies , TB, and Il lustrated 
Book is v±rtually the same. 

C. AltErnate Texts. While the pri^ncipial text for this course is 
Halliday & Resnack, Physics for Students of Engineering and Sci- 
ence , collateral text for references ara provided on the Study 
Guides- Copies of all other texts mentiDned are available for 
use during the classroom periiod. Secauste of the limited supply, 
however, they-inmst be returned to tihe su^jply room at the conclu- 
sion dcff class- 

D. Class InsftTuctar, One of the major differences between 
this Gourse and a conventionally taught course is the role of 
the itfistructor . Your regularly assigned* Instructor will be 
available during all class sessions to assist you with prob- 
lems you encoumter. He wiJll not be le^rtuxing.- You are en- 
couraged to seek help duriiOig -the regular classroom period or 
during an EI period, if yom are having difficulty with the 
course content or mechanics of the cour:se material. 

3. Course Schedule. . Every effort has been made for you to determine 
your own speed in covering course materials. The lExc^ress Checks, 
which will ba mtrntioanBd later in deteail, are taken at the oampletion of 
a number ol Problem mid So!iiutlon Boole segments and luast on a fixed date. 
Obviously, to complete the xrcmirse in the sixtEsen (iSf]i :weeks allowed, 
some minimum rate of pro:gEffi:s naust be maiLntHirred . ZHiis minimum rate of 
progress is determined by t3i6 ^lest areas in tthe foll0wing schedule. 

HTST/LECTTOE SCHEDULE 



9/10 


Oct 


TEST: SEGMENTS 1-10 


23/24 


Oct 


LECTURE : KINEAR MOMENTUM 


6/7 


Nov 


■ TEST: SEQKINTS 11-45 


13/14 


Nov 


LECTURE : JKGULAR MOMENTUM 


4/5 


Dec 


TEST: SEGMENTS 46-52 


11/12 


Dec 


LECTURE : mVE MOTION 


16/17 


Dec 


LECTURE : SUUND 


8/9 


Jan 


LECnURE: G3B0METMCAL OPTICS 


19/20 


Jan 


TEST: SEGMENTS 54-62 



Progress Checks to be :completed -after the following segments: 

5, 10, 18, 46, 48, 52, 58, 62 

Midshipmen who complete all. of the course segments (and appropriate 
Progress Checks) early will be excused from class early except for 
tests and lectures noted above. 

4. Testing . Two types of testing or evaluation are used in this 
course . 
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A. Progress Checks are completed as noted in course schedule. 
Each midshipman will no.tlfy his instructor one period in ad- 
vance when he wishf^.s take a Progress Check. Each Check 
consists of ten questions, similar ta those in the Problem 
and Solution Book, cowering the mateiriLal since the last Pro- 
gress Check. Fif ty, tn±niutes per Check will be allowed. The 
primary purpose of ihe Progress Check Us to identify areas 
where the student is ^\s?^ak. Although they are scored, the 
scores are not used in determining course grades. Scoring 
is as follows: 

Questions Correct Action 

8 or naore ContJmic with next segment 

5-7 Schedule recheck with instructor 

Less than 5 Schedule EI period 

In all cases , midsiiipmen will be prcHridSed references to appro- 
ptl^te cxaoirse materials for the que^tfiojas they miss . The re- 
check will cover only items missed <m tSsss: Progress Check and 
must he schedxiled ini adv*aiice.. 

iB. Di^postic Tests cauivering more TDnatErial than the Progress 
iChecks W!ill :be admiiEESttered as shown mn idbe class schedule. 
These tests will be besciE Jor determiiiia^ progress marks sub- 
mitted to tlae registrar*. Students who hame difficulty main- 
taining Idhe minimum class schedule may fliad themselves takxng 
a test fox grade on. material for which they have not had a 
Progress Check. In these cases, the .-feest raatl also count as 
a Progress Check. Midshipmen who maiaieaML an. A average in 
the four tests will feave the ttest grade ccttiiaitred as the final 
escam grafle. 

5. jLaboratxaary Work . The opescaticm of tiie seiif -paced laboratory is 
oulfiined in a separate. handouC* It is emphasized ::that completed 
Experiment Guides represents tiie work of ±he mMishipman whose name 
appears on the cover. Material, from the laborajEoxy will be covered 
on the final examination. 

6.., : Homework , Homework. problems assigned in the Study Guide are supple- 
mentary. They will not be collected. 

7. -Grades. Progress and semester grades remain the perogative of the 
individual instructor and will be based on: 

Quarterly Diagnostics 
Final Examination 
Laboratory 

Weights assigned to each item will be determined by individual instruc- 
tors. 
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2.3.2.4 Sample Procedure Form 



UNITED STATES NAVAL ACADEMY 
Physics Department 

SP211, Engineering Physics I First Semester, 1970-1971 

The Self-Paced Laboratory 

For the laboratory portion of SP211, Engineering Physics I, you will 
be allowed to proceed at your own pace through the experiments listed be- 
low. Although you have been scheduled for a particular laboratory per- 
iod, you may disregard .this and report to the laboratory (Room B8, 
Michelson Hrll) at any time during the academic day (0755 to 1145 and 
1315 to 1505, Monday through Firday , and 0755 to 1145, Saturday). Upon 
repcrfing to the instructor on duty in the laboratory, you will be given 
an Experiment Guide and directed to a laboratory station for the appropri- 
ate experiment. On the Experiment Guide, you will record your name and 
section J your insturctor^s name, the date, and the times of arrival and 
departure. Upon completing the experiment you may leave, but you must 
give the completed Guide to the laboratory instructor who will, in turn, 
give it to your instructor. Your work will be graded and returned to 
you.- If you wish, you may complete more than one experiment in a given 
week; however you will probably find it advisable not to out--pace your 
classroom assignments. Should you find it impossible to complete the 
experiment in the time available, make a note to this effect on the Guida 
and turn it in to the laboratory instructor before you leave. It will be 
held in the laboratory so that you may return and finish it later* In 
any event, you must complete the experiments within the spans of time • . 
given in the list below. You must complete all experiments or stand in 
danger of receiving no credit for the course. All work is to be accom- 
plished individually, but the instructor in the laboratory will answc..* 
questions and try to ensure that you have adequate' equipment and thsc it 
is working properly. Laboratory Manuals will be provided at each station 
but you should always bring your slide rule, textbook, and a straight 
edge to the laboratory. \ 

EXPERIMENTS 



Available Must be Completed 



Number 


Title 


Beginning 


no later than 


.. 1 


Measurement* I 


14 Sep 


3 


Oct 


2 


Acceleration 


21 Sep 


10 


Oct 


3 


: Measurement II 


28 Sep 


17 


Oct 


4 


Centripetal Force 


5 Oct 


24 


Oct 


5 


' Inclined Plane 


12 Oct 


31 


Oct 


6 


Ballistic Pendulum 


19 Oct 


7 


Nov 


7 


Rotational Inertia 


2 Nov 


21 


Nov 


8 


Vibrating Sr^ring 


16 Nov 


5 


Dec 


9 


Speed of Sound in Air 


30 Nov 


17 


Dec 


10 


Standing Waves 


7 Dec 


9 


Jan 


11 


Single Lens 


4 Jan. 


15 


Jan . 


12 


Lens Combinations 


4 Jan 


20 


Jan 
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STUDENT MANUAL 



!• DESCRIPTION OF THE COURSE 

The self-paced physics course differs from conventional courses in 
a number of ways. It is largely student-managed programmed instruction. 
Most of your learning will be derived from reading carefully selected 
passages in excellent textbooks, simplified written discussions of the 
highlights of the various subject areas, and the use of audiovisual 
aids in lite form of videotapes, "talking books", and brief, meaty illus- 
trated pamphlets called Illustrated Texts. An instructor will be avail- 
able for tutorial assistance as well as diagnosis of your progress. 

The format of the course permits you to monitor your performance 
and achievement by means of instant feedback from the visual response 
mechanism to be described later. 

In addition to self-paced theoretical instruction, you will also 
spend an adequate amount of time in the physics laboratory and attend 
a demonstration-lecture periodically. 

You v/ill always know in advance when a check quiz or an evaluation 
test is to be given. As a matter of fact, you will determine for your- 
self when progress checks will be administered to you. In addition tc 
other periodic tests, a final examination will be used to evaluate your 
achievement. 



2. COURSE STRUCTURE 

Assigned reading — From standard textbooks, coded as follows: 

HR means Halliday and Resnick, PHYSICS FOR STUDENTS OF 
SCIENCE AND ENGINEERING, fifth edition, combined form; 

SZ means Sears and Zemansky, UNIVERSITY PHYSICS, third 
edition, complete; 

AB means Albert Baez, THE NEW COLLEGE PHYSICS - A SPIRAL 
APPROACH, first printing; 

: SW means Shortley and Williams, ELEMENTS OF PHYSICS, fourth 

edition. 

The 'required or prime reading assignment for each segment of the 
course will be identified by one or more asterisks before the chapter 
numbers. The remaining reading is to be considered supplementary. A 
typical reading assignment and its interpretation will be presented as 
a sample later in this Manual. 
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For maximum effectiveness, all assigned reading should be completed 
before you begin work on the programmed Instruction. This first read- 
ing need not be exhaustive because it is anticipated that you will return 
to certain sections of it time and time again as you v;oi'k through the 
segment. 

Infomation Panels — Aside from your textbook reading, much of 
your factual and procedural information will come from Information Panels 
presented in the PROBLEM AND SOLUTION booklet for each segment. These 
Panels are concise discus jions relating to the principles and methods of 
solution involved in the accompanying problems. If you should find that 
you do not fully understand the material in the Panel for a given section 
of your work, you would be expected to return to the textbook assignment 
for clarification. 

Audiovisuals — These are important adjuncts to your reading and 
problem solving. When you are directed to work with a specified audio- 
visual, you will usually be given the option of selecting one of three 
media of presentation. 

Videotape: a demonstration accompanied by a discussion that 
you view on the screen of a small videotape playback; 

Talking Book: a set of carefully constructed pictures and 
diagrams accompanied by an audiotape lecture; 

Illustrated Text: a set of pictuires similar to those used for 
the Talking Book accompanied by a formal written discussion matched page 
by page to the illustrations. 

Progress Checks — groups of relevant questions which you must answer 

after completing a specified number of segments. These checks wi .1 be 

used for diagnosis, progress evaluation, and tutorial assistance should 
the latter be needed. 

Quarterly Diagnostics — special test forms administered periodically 
to assist your instructor in diagnosing possible weak areas in your 
learning pattern, and to enable him to prescribe remedial work where 
required. The Quarterly Diagnostics will also be used to evaluate your 
achievement. 

Final Examination — standard examination which provides information 
relative to your final grade. 

Enrichment Packages — for those students whose progress warrants 
additional, higher level material; to be a student option. 
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3. PRINTED LEARNING MATERIAI.S 

PROBLEM AND SOLUTION BOOK. (Hereafter referred to as the P&S.) This 
is bound study material containing the work for three or more segments in 
a volume. The entire course consists of 45 segments for the semester. 
The P&S material in a given volume will contain blue title sheets between 
segments to help you find the one you want quickly. Each P&S contains: 

(a) A problem section in which the questions and numerical prob- 
lems are presented in strict numerical orJer, to be worked on in se- 
quence . 

(b) A solution sectAnn in which the corrGCt methods of answering 
questions and solving problems are presented in scrambled order. Many 
of these solutions arc terminated by additional "true-false'* questions 
to be answered immediately after you study the individual solutions. 

(c) Information Panels strategically interspersed throughout the 
problem section. 

STUDY GUIDjc:. This is just what its name implies: a written guide 
that you muse follow step-by-step, strictly in the order presented, to 
work your V7ay through the problems, information panels, audiovisuals , 
reading, solutions , and other check points. The remainder of thijs Man- 
ual will be devoted io an explanation of the way in which all these 
aspects of your learning are related. 

4. HOW TO USE THE STUDY GUIDE 

Plti^ase refer to the sample Study Guide which is the If.st page of 
this booklet. It is a partial mock-up of a segment that doesn't really 
exist, and will be used for explanation purposes only. If you are to 
understand how the system works, if you are to avoid blunders when you 
start work on your first actual segment, you must walk through the fol- 
lowing explanation without missing a step. Take your time; be abso- 
lutely certain you understand each maneuver perfectly. If you need help 
in interpretation, ask for it, ... 

Before you begin work on tny aegment, asf:»^r;tain that you have the =^ 
correct STUDY GUIDE by checking thia* .nifmber near the upper right-hand 
corner, then complete the heading on. each STUDY GUIDE sheet. 

Another preliminary step: look at the bottom of the STUDY GUIDE 
sheet and note the number of pages you should have in your hand. Few 
STUDY GUIDES contain more than two pages. Be sure you have what you 
need before you start work. 
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The letter P above the left column means ''Problem Number;" the 
STEPS are also numbered to indicate the sequence of things you must do 
other than problem solving. 

All right. Let's go through the sample. 

Step 0.1 The reading assignment for the segment. The required 
reading is in Halliday and Resnick, paragraphs 49-3 
through 49-6 and paragraph 49-9. The slash-bar (/) 
always means from one paragraph throxigh the other, in- 
clusive. The supplementary reading is in Sears and 
Zemansky, paragraphs 45^6, 45-7, and 45-11. This read- 
ing should be gone through at least once before continu- 
ing. 

Step 0.2 When you have finished your reading, turn to the first- 
page in the P&S for this segment. Read the Information 
Panel, be sure you understand if fully, then continue. 

T This is the first problem in the P&S. Note the overscore 

and underscore lines. These indicate that the problem 
is a core type, required of all students in the course. 
You will find this problem boxed for the same reason in 
the P&S. The problem you find in the P&S as number 1 
is: 



How many gallons of regular gasoline could you have 
purchased with 5 Martian zilches in Septimus, Ohio 
in the year 1960 and still have some change left over? 
t 

A. 1 
•.B. 2 

' C. 3 .^r.^^- 

D. 4 , 



Now Qbviously, to solve this problem you would have to 
know the price of gasoline per gallon in U.S. currency 
and also the equivalent buying power of a Martian zilch. 
Presumably, your reading and the Information Panel con- 
tains this information but let us suppose that you 
didn't do any of the reading and so didn't know the 
answer.. So — you're about to make a wild guess, let's 
say, answer A.r At this point you rub the "reveal" 
pen provided all over the inside of box A for the 
first- question. As you do so, you will see an X appear, 
showing that the selection was incorrect. Do it now; 
reveal the X in box A with your pen. (Best results are 
obtained by rubbing the pen lightly over the surface, 
"then waiting a few moments for the revealed information 
to darken.) 
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Making another stab at it, you choose answer B and use 
the pen bringing out another X. Trying C, you find that 
the pen reveals the characters 29 [a]. This tells you 
to turn to page 29, item [a] in the P&S where you V7ill 
find the full explanation of the method used to solve 
the problem. For this core question, you will also 
find a very short true-false question immediately after 
the correct solution. This question reads as follows: 

A Mart. .an zilch is the oquivalent of three U.S. 
nickels. True or False? 

You must now use the reveal pen on either the T-box 
or the F^box. for question 1. 

If you make the correct true-false selection, a l/will 
appear in the box. If you choose incorrectly, an X will 
appear in the box. The true-false questions are usually 
so simple that you will be permitted few, if any, errors 
in thils part of the work. Getting one of these T-F's 
wrong is a pretty sure indication that you are not read- 
ing the solutions. You must avoid this. 

Let's go down to the next step. 

Step I. 1 You are now being given an option. If your first choice 
was correct, you will be permitted to skip over the next 
four questions and advance to the next Information Panel 
If you answered incorrectly, even once, you must go 
through the remedial loop consisting of questions 2 
through 5. 

We ar^ assuming that you missed question 1^, so let's 
go through this loop together. 

2 Problem 2 in the P&S. It is not scored, hence it is 

not a core problem. It reads as follows: 

It is predicted *that a gallon of regular gasoline 
will sell for $1.05 by tha year 1998. If this is 
roughly 3-1/2 times the price of gasoline in 1960, 
how much Jid one gallon cost in 1960? 

This is not multiple-choice, It*s a completion type of 
question where you must write in the answer. So, write 
your answer on the line below the rectangle for question 
2. The answer is, of course, 30c because $1.05 is 3-1/2 
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times 30C- After writing it in, reveal the answer in 
the rectangle with the pen; the answer 30c will appear 
accompanied by the referral page and item 14[c]. Turn- 
ing to the referral, you find the solution worked out 
for you to check your own thinking. Problems that are 
not core types are not accompanied by true-false check 
questions, so you're ready to go to question 3. 

Let's interrupt the sequence for a moment. Kven if you 
were able to answer the original core question correct\y 
the first time, you should go through the remedial loop 
anyway if you have any doubt at all about the method of 
solution or the answer. You may have guessed at the 
right answer, or you may have made two errors that can- 
celed out. In any case, if you fe^:! that your choice 
of the right answer was a fluke in any way, we urge you 
to go through the remedial loop. 

3 Problem 3 in the P&S ; it is not a core problem. Here 

it is : 

Ten Martian zilches \^ill buy exactly the same number 
of 2-1/2 inch Mcintosh apples in a given market on a 
given day as two U.S. dollars. Thus, one zilch is the 
equivalent of 

A. 100 

B. 20<? 

C. 40C 

D. 600 

A glance at the STUDY GUIDE corroborates the fact that 
this is another multiple-choice question. Apparently 
10 zilches is the equivalent of $2.00, so one zilch 
must be worth 200, This is answer B, so if you use the 
reveal pen in box B you will bring out the instruction 
18 [b] indicating that page 18, item [b] in the P&S has 
the solution. I^ether you wer<a right or wrongs in your 
selections, it is important that you read and iunders tand 
the solution. If you had chosen any answer other .tihan 
B, you would have revealed an X as before. Tbere> is no 
true-false question, hence you can now go-on to. question 
4. 

Here is your first modified true-false question: 

True or false? Five martian .zilches will purchase 
more milk than 20 U.S. dimes. 
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Note the italicized word. Read the statement and (a) 
if you decide it is true, simply ink the T-box on 
the STUDY GUIDE; (b) if you feel that it is false, 
write a x^ord that can replace more and thereby make 
the statement true. After you have written the correc- 
tion word on the line under tlie F rectangle, then, and 
only then, you are to reveal the answer with the pen. 
In this particular instance, the correct answer is 
''false" and you would write in the ^-jord **les5" in place 
of more. Your reveal pen will bring this out, too. 
If you had selected "true" as your ansvar, the pen 
would have revealed an X inside the T-bo::- So, after 
writing "less" you would see revealed: "less (21[d])." 
At this point in an actual lesson,- you would turn to 
this page and item in the P&S and read it carefully 
before continuing the sequence. 

Continuing with the remedial loop: 

5 Another multiple-choice question: 

In order to have filled your 18-gallon tank with 
gasoline in 1960 in Septimus, Ohio, you would have 
spent at least 

A. 15 zilches 

B. 21 zilches 

C. 23 zilches 

D. 27 zilches 

The correct answer is, of course, 27 zilches since each 
zilch is worth 20c and each gallon costs 30c , so you 
would reveal box D and find inside the instruction 
"27[b]." After reading the solution, you again encounter 
a check T-F question which is then answered as before by 
revealing either the T or F box in question 5. Any 
answer other than D above would have revealed an X just 
as described for the previous multiple-choice question. 

Step 5.1 Everyone is now expected to devote some time to- '.the: 
Information Panel, "The Currency of Venus" and then.- 

Step 5.2 select the medium he wants for running through the audio- 
visual, COINAGE AND BILLS OF THE INNER PLANETS. 

After that is finished, everyone star_ts once again on an 
equal footing with the core question 6. 

And so forth. 
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STEP 



STUDY GUrOE 



SELF-PACED PHYSICS 



NAME 



STEP 



SECTION 



SEGMENT AO 



0.1 



0.2 



Reading: HR*49-3/49-6 ; *49-9 

SZ 45-6, 45-7; 45-11 

Information Panel, "The Currency 
of Mars" 



B 



□ □ 



1.1 



3 



5.2 



6.1 



If your first choice was correct 
advance to 5-1; if not, continue 
sequence. 



(ans) 



(ans) 



u I r 

□ □□ 



Information Vnne'l , "The Currency 
of Venus" 

Audiovisual, COINAGF AND BILLS 



OF THE INNER PLANET ^ 



■ (ans) 

If your answer was correct, ad- 
vance to 9.1; if not, continue" 
with sequence. 



(ans) 



Note: In this sample "waiu 
through", we have not in- 
cluded the Information 
Panels nor any set-up ? & H 
The probi "ims that would noi 
mally appear in the P & S 
are given in the Student 
Manual for explanation pur- 
poses . 



AND SO FORTH 
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2,4 Infomation and Data Handling Procedures 

A complete definition of the course content is necessary and ' 
was accomplished by the identification of over one thousand MBO^s. These 
are to be found in the two volume MBO docimientation . 

2 • 4 . 1 Dqta Preparation Prior to Course Implementation 
2. 4,1.1 Establish Course Structure 

1. Student Background and Knrollment rile - Prepare 
a student background card for each student to be enrolled in the course. 
See Exhibit "A'' for data and format. Under CMl , there j,s an allowable 
maximum of one-hundred and eighty-five students per CMi course. 

Purpose - This data registers a student in the 
course and provides the basis to accomplish the following: 

a) Assignment of Course Student Number 

b) Creation of StudGnt Roster 

c) Comnutcition of Capability Index 

d) Storage of Background . Data 

2. Preparation of Course Header File - Prepare a 
course header card (Port-A-Punch format) for each test (or Study Guide) 

to be administered during the course. See Exhibit "3" for data and format. 
There is an allowable maximum of 48 questions per test. 

Purpose - This data establishes the correct answer 
matrix for each test to be computer graded and provides certain' lesson 
identification data. 

3. Preparation of Question (MBO) Description File - 
Prepare a question description card for each question within a test (or 
Study Guide). See Exhibit "C" for data and format. Following the basic 
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structure of the course, each question should be designed to test the 
student's knowledge of a specific Terminal Objective (and Enabling Objec- 
tive where applicable) . 

Purpose - This data will be used in the prepara- 
tion of the individual student performance report (Refer to Exhibit "D") 
and the TO analysis reports (as described later) . 

The prescriptions (remedial assignment for a 
student) for each question rshould be carefully constructed and will be 
printed for the student only when the question is answered incorrectly. 
See AIMS III Systems Manual for more detailed information on these files. 

Non-CMI Procedure - If the CMI is not utilized 
for generating prescriptions, then a TO Key Sheet should be prepared 
as in Exhibit *'E." These sheets are issued subsequent to each Quarterly 
Diagnostic. Section 4 contains a detailed description of this procedure 
and tiie Progress Check procedure. 

2.4.1.2 Data Verification and Updatin g 

Once the data files have been prepared, CMI reports 
are generated to reflect what data has been entered into the system. 
These reports should be reviewed to verify accuracy of the data. 

1. Student Background and Enrollment 

Print3d reports consist of the Class Roster 
(Exhibit "F") and Course Background Roster (Exhibit "G") . 

The Course Student Number (CSN) is created by 
the system and is printed on the Class Roster. The CSN is the identifying 
number to be used subsequently on all student response input data. 

To add a student to the file, prepare a student 
background card as above; to drop a student from the file, prepare a 
student drop card as in Exhibit ''H." 
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2. Course Header 

Certain types of errors input to this file, will 
automatically be printed by the CMI; however, the temporary (or 
Permanent) Header File Listing (Exhibit "I") should be reviewed for 
completeness and accuracy. The AIMS III User's Manual contains detailed 
instructions on the method of updating this file. 

3. Question (MBO) Description 

The Course Structure Summary Listing (Exhibit "J") 
provides a liue-by-line printout of the MBO descriptions. Review of the 
listing is necessary to verify all question description data. Additional 
Information is contained in the AIMS III User's Manual, 
2.4.2 Course-in-Progress Procedures 
2.4.2.1 Student Data 
1. Data Preparation 

Under CMI, two forms of collecting t^st data are 

possible: 

a) Port-a-Punch and 

Optical scanning form 
See Exhibits "K" and "L" for Port-a-Punch formats 
and descriptions* See Exhibit "M" for Opscan data. 

. If the non-CMI procedure is utilized, the Opscan 
£brm is also utilized but as described in Exhibit "N." 

Note that "completion time" data is an optional 

data requirement . 
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Responsibility 
STUDENT 



INSTRUCTOR 



COURSE MANAGER 



DATA MANAGEMENT GROUP 



Progress Check Procedure 

Procedure 

1. Schedules Progress Check (PC) with instruc- 
tor upon completion, of appropriate segments • 



2. Submits Study Guide to 



1. Reviews student's Study Guide just prior to 
PC. 

2. Administers PC to student. 

3. Grades PC ''on-the-spot." 

4. Takes following action: 

a) if 40% or less: o^mfee-t: schedules 
t#?ta^^#l sess^'GU^ ^ ;.^.^®ties remedial 
5itud^in:i mLll ^retake PC. 



U> issues remedial material; 

'may reschedule PC. 

c) if 80*100%: issues remedial material: 
n^ St^y <^lde. 

NOt^i: Instructor Option: 

Examines Study Guide as aid to diagposis 
of student weakness; may conduct i*^JU3cM- 
ate tutorial if warranted, 

I 5. Issues a "filled-in" Study Guide to tba 

student for each one he turns in finally. 

6. Forwards PC responses to course manager. 

1. Forwards graded Progress Check to Data Man- 
/ agement Group. Do not include "retakes." 

1. Prepares Progress Check worksheet (See f^^l- 
lowing sample.) 

2. When approximately 33% of all students have 
taken the PC, prepares Progress Check-sum- 
mary report (See following sample.) 

3. Forwards summary report to course manager. 
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2. Progress Check Procedure (cont'd) 
Responsibility Procedure 
COURSE MANAGER 1. Reviews summary report with instructor. 

2. Determines remedial strategy. 
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PROGRESS CHECK SUMMARY REPORT 

NAW PHYSICS - PS211 

SEGMENT 



SAMPLE PROGRESS CHECK - SUM>URY REPORT 



DATE 



X TAKEN 



% AQIIEVEMEOT 



CORE PROBLEMS 



SECTION 



0101 



0301 



0501 



0801 



1101 



1201 



AVERACK % 
ACHIEVKMKNT 



I 



Total 7. 
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SAMPLE PROGRESS CHECK WORKSHEET 



3. ■ Quarterly Diagnostic Procedure 



Responsibility 
COURSE MANAGER 



INSTRUCTORS 



DATA MANAGEMENT GROUP 



COURSE MANAGER 



INSTRUCTORS 



Procedure 

1. Prepare a random question Diagnostic for 
each section, consisting of one question 
for each TO, dra™ from a Multi-Version 
Question Bank. 

2. Prepare sufficient copies of Student TO 
Key Sheets. 

3. Prepare sufficient copies of each Diagnostic. 
A. Prepare correct answer key. 

1. Administer the' Diagnostic. 

2. Fon^ard the Student Diagnostic Answer 
Sheet to the Data Management Group. 

3* Prepare make-up Diagnostics, administer 
and grade them. 

1. Sort Diagnostic Answer Sheets into sections. 
(See following procedures for items 2-5) 

2. Prepare Student TO Key Sheet (one per stu- 
dent) for Diagnostic. 

3. Prepare Diagnostic TO Summary Report for 
each section. 

4. Prepare Diagnostic Score distribution (Histo- 
gram) by section. 

5. Prepare Diagnostic T.O. Summary Comparison 
Report. 

6. Forward TO Key Sheets and Reports to 
course manager 

1. Distribute appropriate reports and TO Key 
S%eets to instructor. 

2. Review Comparison Reports. 

1. Distribute TO Key Sheets to students. 

2. Review reports to determine remedial strategy 
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SAMPLE procedurf: 



Procedure for Marking 
Student TO Key Sheet and TO Section Summary Worksheet 

!• Post student's last name and Academy // to TO Key Sheet from 
his Diagnostic Test. 

2. Review each Student Diagnostic Test Sheet to determine incorrect 
answers . 

3, a. For each incorrect answer, use cross-preference list to 

determine appropriate TO; circle that TO on the TO Key 
Sheet. 

b. For each incorrect answer, post a check mark to the TO 
Section Summary Worksheet, 

■ , ^, Total wrong 

l^. Calculate and pose I wrong = xotal students this section taking test 

5. Also post % wrong to the Diagnostic TO Summary Report for the 
appropriate section and the Diagnostic TO Summary Comparison. 

Total wrong answers for TO 

6, Calculate % wrong - all sections = ^^^^^ students taking test 

and poc?f. to Diagnostic TO Summary Comparison. 
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SAMPLE PROCEDURE 



Procedure to Prepare Diagnostic Score Distribution 

!• Count the number of correct answers on each student test sheet 
. and post the # correct on upper right. corner of his test sheet, 

2. Iflien all student tests have been marked as in (1) above, complete 
the Score Conversion Table .as follows: 

a. Place a mark in the check box for each student 
opposite the appropriate "// Right'* line; 

b. When all student tests have been checked as above, 
post the total for each line in "Total" column. 

3. Draw Bar Chart for section on Score Distribution Form. 

4. Post totals from "Total" column to "All Sections" Score Table. 

5. When all sections have been posted as in (4) above, add and post 
• totals to "Grand Total" column. 

6. Draw Bar Chart for all sections on each Score Distribution Sheet. 
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SAMV^LIC FORM 



T 



SCORl' CONVERSION TABLK 



•J- 
I 



DATE 
SECTION 



// RIGHT 


(inclusive) 


SCORE 




TOTAL 


31 




100% 






28 - 


30 


90 


• 




25 - 


27 


80 






22 - 


24 


70 






19 - 


24 


60 






16 - 


18 


50 






13 - 


15 


40 






10 - 


12 


30 






7 - 


9 


20 






4 - 


6 


10 






0 - 


3 


> 10 
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■ SAMP1..E FORM 

DIAGNOSTIC T.O. SUMMARY RKPORT 
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T.O. SKCTIOM SUJIMARY WORKS.U-.ET " SAMPLE FORM 

DATE 



SECTION Total 

Students 
This Section 
Taking Test 



T.O.// 



CHECK 
WRONG 



TOTAL 
WRONG 



WRONG 
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SAMPI,!'; FORM 



QUARTEllLY DIAGNOSTIC - SCORE DISTRIBUTION 



NAVY PHYSICS - PS211 



120 — 



100 — 



W, 80 
H| 

Wj 60 



Q 



HI 40 



20 -- 



ALL SECTIONS 



DATE 



>10 10 20 30 40 50 60 70 80 90 100 



SECTION 



25 -A 



ERIC 



Wl 20 
col 10 



5 - 



0 — 



>10 10 20 30 40 50 60 70 80 90 100 

PERCENTILE 
70 



2.4.2.1 (ContM) 
4. Study Guide 

For a detailed listing of the multiple responses 
selected by all students, an item analysis report is available. Under 
CMI, the report is prepared via the computer and is described ±]\ the 
AIMS III User's Manual. Descriptions for the course revisions are con- 
tained in the Empirical Course Development Model and the Revi.r 'ons Pro- 
cess Documentation . 

Under CMI, Study Guide responses are also used in 
the determination of a student's understanding of a specific TO. 

2.4.2.2 Report Descriptions 

The non-CMI reports have been described in the pre- 
ceding sections. The student oriented CMI reports available are briefly 
described below and are also contained in the AIMS III Systems Manual, 
and incorporated in this report as : 



Exhibit "0' 



Exhibit "P' 



Exhibit "Q' 



Exhibit "R" 



Exhibit "S" 



Individual Student Analysis; student^s per- 
performance on each individual TO. 

TO Remedial Summary; listing by TO of 
all students who performed above the 
error percentage set for a particular 
TO. 

Remedial Session Lesson Plan; TOs as- 
signed to a session (and instructor) 
according to the number of students who 
performed above the error percentage. 

Remedial Session Roster; a listing of 
students assigned i:0 remedial sessions 
based on their TO error perce:it£ige. 

Volume Summary; a listing of all the 
total perf orraar.ce for all students by 
section. 
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EXHIBIT "A'' 



STUDENT BACKGROUND AND ENROLLMENT CARD 
Column Number Heading . 



1 


-25 


Student Name 


26 


- 34 


Academy Alpha Number 


35 


- 37 


SAT - Math 


38 


- 40 


SAT - Verb 


52 


- 53 


Calculus 


54 


- 55 


Chemistry 


56 


- 58 


Math Achievement 


61 


- 63 


Rank 


64 


- 78 


Cormients 


78 


- 80 


Course Number (*) 



(*) NOTE 

If this number is input as zero, the system assigns the student 
a course number; if not, it considers this card an update. 
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KXIIIBIT ''B" 



COURSE HI-ADER CARD 



Column Number Heading 



2, 6, 8 


Number of Selections 


24, 26 


Nimiber of Questions 


42, 44 


Course Number 


48, 50 


Sequence Number 


54, 56 


Type Number 


60, 62 


Volume Number 


6i^, 68 


Segment Number 


80 


Single or Double 
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EXHIBIT "C" 

QUESTION (MBO) DESCRIPTOR CARD 

Column Numbers Heading 

1, 2 Volume Number 

3 Segment Number 

4, 5, 6 TO Number 
7,8- . EO Number " 

9-42 MBO Description 

43, 44 Learning Category 

48 Type 

49, 50 Question Number 

51 Correct Answer 

52 - 78 Prescription 

79, 80 Course Number 
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EXHIBIT "E" 



STUDENT TO KF.Y SHEET 



SEGMENT NUMBER - CORK PROBLEM NUMBER* 



NAME 



ACADEMY NUMBER 



TO 


SEG. // ■ 


- CORE PROB. 


TO 


SEG. // 


- CORE 


1 


1-13 




16 


4-16 




2 


2-17 




17 


5-5 




3 


1-1 




18 


6-1; 


6-2 


4 


2-1; 


2-6 


19 


6-9; 


6-14; 


5 


2-10; 


2-14 


20 


7-2 




6 


3-1; 


3-6 ; 3-9 


21 


7-1; 


7-5 


7 


3-1; 


3-6 ; 3-9 


. 22 


7-10 




8 


1-13 




23 


7-15 




9 


3-1; 


3-6 


24 


8-1 




10 


3-17 




25 

1 

26 


7-18 




11 


2-14 




8-9 




12 


3-12; 


3-18 


27 


8-5; 


8-18 


13 


4-1; 


4-2 


28 




9-2 


14 


4-5; 


4-6 ^ . 


29 


9-3 




15 


4-11 




30 
31 


10-1; 
10-13 


10-5 


*CorG 
The 


Problems 
TO number 


Most Closely Keyed to 
circled was answered 


1 Diagnostic Questions, 
incorrectly on the Diagnostic 
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EXHIBIT "H" 



STUDENT DROP CARD 



Coluina Number 
1-20 
21-77 
78 - 80 



Student Name 
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EXHIBIT "L" 



PORT-A-PUNCH STUDENT REf.PONSE CARD 



Column Nuniber 
2, 6, 8 

12, 14 
18, 20 
24, 26 

3u, 32 

36, 38 

42, 44 

48, 50 

54, 56 

60, 62 

66, 68 

72, 74, 76, 78, 80 ' 

Shaded Columns 

Between Columns of Data 



lleading 
CSN (Course Student Number) 
Date 

Month 

Day 

Year 
Completion Time 

Hours 

Minutes 
Course Number 
Sequence Number 
Type 
Volume 
Segment 

Student I , D . Number 
Responses 



The student's answer to each question is punched in the shaded column 
the card. 
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EXHIBIT "M" 



Student Response OpScan Form 

DATA CONTENT 

Student Name 
Data 

Social Security Number 

Course Number 

Completion Time 

Volume 

Segment 

Type 

Page Number 



The student fills in the answer by marking 
the appi jpriate box (multiple choice) with 
dark pencil. 
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EXHIBIT "N" 



INSTRUCTIONS FOR CONPLKTING 
STUDENT DIAGNOSTIC ANSWER SH cET 

(STUDENT RESPONSE) 

(Sample attached) 

1. Print name in space provided (last name first). 

2. Write date in space provided. 

3. Write Academy student number below name. 

4. Write Section Number below data. 

5. Familiarize yourself with the answer sheet. Record your answei's 
in pencil by placing a mark in the appropriate box. Note that 
only the first 31 responses apply to this diagnostic. 

6. If the multiple-choice answers are A, B, C, D, use box 1 for A; 
box 2 for B, etc. 
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3.1 



SYSTEMS DESCRIPTION 



SYSTEM M/\NUAL 



Annapolis Evaluation 



I . INTRODU CTION 

This document is iiitGnded to describe the structure and opera- 
tion of the Evaluation System generated for use with Annapolis 
data files. Hhe. system has been designed to isolate, subject 
to constraint, desired variable subsets and to perform a wide 
range of statistical analyses and tests pertinent to evalua- 
tions of experimental results. 



The Evaluation system consists of three major components. 

Data Files - containing all information generated 
in the course of the experiment. 

Accessing Routine - Program for generating any - 
constrained subset of data from the files. 

Statistical Routine - Permits user selection of 
desired statistical tests and summaries pertinent . 
to data evaluation. 



All information gathered in the course of the experiment is 
contained in eight master data files. The structure of these 
files is listed in Table II. 1 - II. 8 as well as array dimen- 
sions needed to write out the fields and frequency of recording 
data (See Appendix 1) . 



II. 



STRUCTURE 



II. 1 



DATA FILES 
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11,2 ACCESSING ROUTINE 



11.2.1 OPERATING CAPABILITIES 



This segiT/cnt can be defined r,s the interface between the user, 
the data files, and the statistical componenua of the system. 
Its purpose is to retrieve information from the files to the 
degree of specification or constraint required by tlie user. 
The system is able to: 

A. Retrieve the observations stored on any one or 
more than one requested variable. These requested 
variables may be of : 

i. The same type; i.e. 5 student average con- 
fidence for posttest A, for posttest C, 
and for posttest I — here student average 
confidence is the variable and there are 
three variables of the same type requested 
for each posttest. 
li. Different types; i.e., retrieve average 

student, confidence and the average student 
log confidence for posttest A — here there 
are two variables, average student confi- 
dence and average student log confidence. 

B. Treat all constraints on variables sequentially in the 
order that they are imposed by the user. There are 
two types of constraints, descriptive constraints 

and limiting constraints. (The descriptive con- 
. straints should be specified first sequentially by 
the user.) 
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i. Descriptive constraints are constraints 

necessary to define the proper subset of the 
desired variable, 
ii. Limiting constraints are necessary for iso- 
lating or partitioning the subset of the 
desired variable; 

i.e., retrieve the posttest scores of the 
audiovisual group, for posttest D, for stu- 
dents who scored under 80 *for that test and 
where IQ was in the range of 100 to 130 and 
whose SAT Math marks were under 100, 
In this example the variable is posttest scores. 
There are two descriptive constraints, the 
group of students (audiovisual) and the par- 
ticular posttest (posttest D) and three limit- 
ing constraints, the actual score for the test 
(only students who scored less than SO) , the 
students IQ and the SAT Math scores. 
The maximum number of constraints that can be placed on 
a particular variable is twenty (20) . Any combination 
of descriptive and limiting constraints which add up to 
twenty or less will be acceptable. 
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II. 2.2 USER INPUT 

The following is a definition of user input required for 
retrieval of data 

X - the quantity X is the variable under 

consideration, for example, post scores 
or average student confidences 
- where, i can go from 1 to 20, The quan- 
tity is the i^'^ parameter 
either limiting or descriptive, to be' held in constraint 
in a definition of the group X. 

'^li ~ lower limit for the i^h constraint 

i 

^2i ~ upper limit for the i^^ constraint 
such that 

^li 1 Zi 1 ^21 
For each variable, the user must provide a variable 
definition card containing the numerical value of X 
and alphanumeric information which will be a defini- 
tion of X. For each constraint (limiting or descrip- " 
tive) to be placed on that variable, the user must pro- 
vide one card containirag the numerical code for the Z± 
and the upper and lower limits under which will be 
held, Aj^j^ and 'A2j[ and alphanumeric information descrip- 
tive of the constraint. 

The above definitions are straightf orv-zard whea the user 
requests a single set of data. When, however, he w.ishes 
to retrieve more than one group, each group must be separ- 
ately defined by a variable definition card and a set of 
constraint cards pertaining to that group. 
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II. 3. STATISTICAL ROUTINK 
II. 3.1 OFERATIN^G CAPABILITIES 

It is the pui'pose of this portion of the system to per- 
form required statistical tasks on the accessed data as 
specified by user input. In the performance of these 
tasks the system will 

A - set an output "Flag" if the user re- 
quests a listing of the accessed data 
B - compile a list of statistics (i.e., mean, 
standard deviation, range, etc.) to be 
computed in the statistic generation 
routine. The sources of this list are 
the direct and indirect commands by the 
user. 

i " The direct commands require a card 
input by the user identifying each 
statistic to be computed, 
ii - The indirect commands result from the 
user specification of a statistical 
test to be performed (i.e., if the 
user determines that a "t" test should 
be performed on two sets of data then 
the means, standard deviations and sample 
sizes of each of the sets are computed 
and entered in the call statement for the 
"t" test subroutine. These three statis- 
tics will then be compared to the existing 
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list of statistics and added to that 
list as required) . 
C ~ Call the statistic generation routine, indi- 
cating in the call statement the statistics 
to be computed and whether or not a listing 
of the data is requested by the user. 
D ~ Call each of the requested statistical sub- 
routines, setting in the call statement the 
magnitude of the statistics which have been 
computed for that test. 
II. 3-2 USER INPUT 

There are three types of user input required for the selec- 
tion routine : 

A. A zero-one indication for "no-listing" of the data 
that is being processed. 

B. A list of all statistics to be computed (this is 
required only for direct statistic specification 
described a^bove) . Table II. 10 contains list of 
possible statistics that can be computed for a • 
single variable, and Table 11.11 those statistics 
applicable to two variables. 

C. A list of statistical tests to be performed on 
the accessed data. Table 11.12 contains a list 
of possible tests, as well as a summary of in- 
ternal quantities calculated for each test and 
transmitted to the particular test subroutine. 
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TABLE 11.10 " STATISTICAL COm^UTATIONS 
- SINGLE VARIABLE 

1 " Count -: to count the number observations in a set 

2 - Sum of the observations 

3 " Sum of the squares of the observations 

4 - Mean value of the set 

5 -'Variance of the set 

6 - Standard deviation of the sat 

7 - Minimum and maximum values of the set 

8 - Range of observations in the set 

9 - Ranks of each observation in the set 

TABLE 11.11 - STATISTICAL COMPUTATIONS 
^ TWO VARIABLES 

1 - Sum of crossproducts of two variables 

2 - Correlation coefficient between two variables 
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t 



TABLE 11.12 

AVAILABLE TESTS AND ASSOCIATED 
INTERNAL STATISTICAL COMPUTATIONS 



TEST DESCRIPTION 



STATISTICS TO BE COMPUTED 



ADDITIONAL DATA 
TRANSMUTED 



Student's t Test 



Sample size 
Mean 

Standard deviation 



None 



F Test 



Sample size 
Standard deviations 



None 



One-Way Analysis 
of Variance 



No. of groups to be 

tested; for each group; 



sample size 
sum of observations 
sum of the squares of 
observations 



Wilccxen signed 
Rank Test 



Sample size for each 
group. 



Set of observa- 
tions to be 
tested 



Kendall coeffi- 
cient, of concor- 
dance 



Chi-Square Test 



No. of groups 
Sample size of each group 
The rank of observations 
within each group 

No, of groups 

No, of cells in each group 
No. of observations in 
each cell 



Cochran's Test 



Sample size 
Standard deviations 



Least Significant 
Difference Test 



Sample sizes 

Means 

Variances 



1 
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II. 3.3 STATISTICAL SUBROUTINES 

The following part of Section 3 contains the detailed specifications 
for each of the statistical subroutines listed in Section 3.2. The 
specification includes required input, computations made within the 
subroutine, and the output. In addition, a reference text for each 
statistical test is included. 
(1) Student's t test 

Reference: John E. Freund , Mathematical Statistics ^ Prentice Hall, 19 

To test the hypothesis that two sampled means are equal 

Input: 

(x^ , x^) the means for the two samples, 

(^l^y ^2^^ variances for the two samples, and 

(n^, n2) the number of observations in each of the two samples. 

Computations: 

Degrees of freedom of the test = n^^ + n2 ~ 2 

The t statistic = 



Kni - 1) Si2 + (n2 - D fi f" 

+ n^ - 2 Y^i ^2 



Output: 



The summary statistics (mean, variance, and sample size) for each 
of the two samples: 

Degrees of freedom of the test. 

• The t Statistic. 

^ test - for testing the hypothesis that two variances are equal 

Reference: Same. 
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Input : 

(Sj^, ^2^^ variances for the two samples, ^nd 
(n^ , n^) the number of observations in each of the two samples. 
Computations : 

Maximum (s,^ , ^2^^ 

• F statistic = — 

Mininum (s j , S2 ) 

Degrees of freedom of numerator = n^ - 1 , whetc i is equal to one 
or two, corresponding to x^hich of the two sampled variances is the 
largest . 

Degrees of freedom of the denominator - - where i corresponds 
to the smaller of the two. 
Output: 

The summary statistics (mean and variance) of the two samples* 

• The F statistics. 

• The degrees of freedom of the numerator and da^^ominator . 
(3) One-way analysis of variance program 

Reference: Johnson & Leone > Statistics and Exper ll Pental Design , John 
Wiley, 1967. 

Input : Each of the h vectors to be compared 

t«l t«2 t^h 



i « 1 

i « 2 

u 

1 - n 
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Compute : 
• = SQRT 



Xj^. wliere x 



i = 1 



/ 



h n 
GTI ^ I 



T . T ti 11 12 hn 

t = 1 1=1 



• G = (nh)"^ 



• GT = (n)-^ 



h n 
E EX 
t = 1 i = 1 



1 2 



ti 



h n 
E EX 
t = 1 i = 1 



ti 



In 



hi 



Output : 

• Listing of data in coluwuis , listing of moan and standard deviation 
each column at the bottom of each column. 

• Table of the analysis of variance as follows: 

One-Way Analysis of Variance of the Measured Parameter 



Sum of. 



Degrees 



Mean 



Source 



Squares Freedom Square 



F-Ratio 



Total GTI-G nh-1 

Between Groups GT-G h-1 

Within Groups GTI-GT nh-h 



GT-G 
h-1 

GTI-GT 
nh-h 



GT-G 
h^l 



GTI-GT 



nh-h 



(^0 The chl-square test - for K independent samples 

Reference: Siegel, Non^-Parnmetric Stat is tics , McGraw-Hill, 1956 • 
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Input: 

K = No. of columns (no. of independent samples) 
R « No. of rows (no. of categories) 
X = a KR matrix with elements 

[Xj^j ] = No. of obseirvations falling into the i^^^ category of the j^^ sample 

NOTE: Xj^j is also r.alled the observed frequency per cell 

Compute: 

• Degrees of fredom = (R - 1) (K - 1) 

K 

• Each row sum RSj^ = I X. . 

j = 1 ^ 



• Each column sum CSj = E Xj^j 

i = 1 



Sum of all rows or columns 

R K R K 

TS = E RSjL = E CS. = E E Xij 

i = l j = l ^ i = 

• Compute expected frequency per cell (E^^j) 



(RSj_) (CSj) 
^ij " TS 



• Compute the chi-squaijE^TBeart statistic 
i = 1. j - 1 ^^J 

Output: 

No. of independent samples (K) 
No. of categories per sanrple (R) 
Degrees of freedom 
Chi-square test statistic 
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(5) Kondall. coefficient: of concordance - to compute the concordance 

(agreement) between K rankings of a set of information 
Reference: Siegel, Non-Parametric Statistics , McGraw-Hill, 1956. 
Input : 



of observations, Tlie nuiul' • of observations in each vector must be equal. 
N = No. of subjects ranked 
K = No. of different rankings 

NOTE: The numerical value of the elements will probably be something 
other than ranks (i.e., test scores). 
Computations : 

Convert each of the column vectors into a column vector of ranks; i.e., 
if X^j (i^^ element (row) in the j column) is the minimum value in that 
column, then it :2jS assigned a vmih ^ r^j = 1. 

If X^A is the masiimum value in tiae^ column, then it is .assigned a rank, 



The remaining rrifes for ranking arrd: for settling ties jare exactly the 
same as for theHSlcoxen signed rank test. 

NOTE: Each colnnm must be ranked independently of the other columns. 
The resulting maanrix (noted R) is 



X-matrix - which can ^ - 



a set of K vectors of observations or ranks 



= N. 



R 




Where r^j is the rank of 
X^j - the rank of the i^^ 
element in the j^^ column. 
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CompuCG : 



B-. = 
J. 



K 

Z 

j = 1 



N 
Z 



Compute : 

2:(t3 ^ t) 



and (A^)' 



T. = 
3 



12 



t = no. of observations. -in a group tied for a given rank* T^ie compu- 
tation of is best exijlained through an example: 





3 = 1 


3 = 2 


d - 3 = 


= .1 


1 


2.5 


t 


2 


4.5 


1 


1 


3 


2 


2.5 


4.5 


4 


4^ 


4.5 


4.5 


5 


3 


.4.5 


4.5 


6 


7.5 


8 


4.5 


7 


6 


9 


8 


8 


9 


6.5 


S 


9 


7.5 


10 


8 


10 = N 


10 


6.5 


10 



Mot 3 = 1 there are two sets of rank ties , each containing two ties, 
I.e.,, 2 - 4.5's 
2 - 7.5's 



Therefore, T-j^ = 



(23 - 2) +(23-2) 



12 



= 1 
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! 

1 

T 

T 

Therclore,, ^ — ■ — ^2 

f 

,J For j = J iiraa33e are two sets of rank ties, 



r 



I 
1 

IerIc 



For j = 2 aiassire are three sets of rank ties, i.e., 
2— 2.5's 
4.5's 
ZL — 6.5's 

Therefore,, (2^ - 2) f- (ji - 2) f- (23 - 2) ^ , 3 



4-— /4.5's 
1:3— E8's 



Therefore-, ^2 



_ - 4) + - 3) _ _ 

in — / 



Compute JL Tj 



EE 

1J 



^ = 1 

• KendaHnUrsBHk :C£n^f;fic±cnt of concordance 

• N = s 

j = 1 

1 

• The test: s^^i^foiiiStic is the chi-square 



chi-SMr^ri- tsst ^atistic (C) = KCN - 1)W 
degi2E5££=^ an' fireedom (DF) = N - 1 
Output / ' 

• M phaHpr^T^ryr^^ of the 'data 

• Listiti^xuftxiiie Si^pat imatrix 3! 

• Listing rf i&tt^ r^fflfeed matrix R. and the computations R^ , , and 
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I 





J = 1 


2 


3 


• • • 


K 






4 — 1 
1=1 


^11 


^12 


^13 






H 


A 


2 


^21 








• 


m. 


• 


• 
• 


• 
• 


• 












• 

N 




• • 


• 




• 


• 

% 


• 


T, 

J 


h 


T2 













• Kendall rank coefficient of concordance (W) • 

• Chi-square test statistic (C) 

• Degrees of freedom (BF) 

(6) Wilcoxen matched pairs signed rank test 
Reference: Siegel, Non-?arametric Statistics * 
Input : 

Two vectors (sets) of numbers: 

Set A Set Ba 



X 



11 



^1? 



X 



2X- 



I 
I 
I 
I 



Computations: 

Count the number of items in each set, 

n^^ = no. in Set A; n2 = no. in Set B. 
NOTE: If ni = n2 set n^^ = 

If tij^ 7^ n2j terminate the subroutine vNii : the following message 

and output : 



ERROR MESSAGK ; '»SAMPL1^S A AND B ARE NOT OF EQUAL LENGTH - WILCOKEN SIGNED 
RANK TEST REQUIRES A COm^LETE SET OF MATCHED PAIRS QF DATA" 

• Compute the differences (d^) where d^ = X-j^^ - 

• Compute the absolute value of differences |d^| 

• Numerically order the |d-|^| by magnitude 

• Compute dummy ranks by assigning 0 when \dj^\ =0 and by assigning 
the numerical count to the remaining ordered |d-|^| > G 

• Compute actual ranks via ithe following rules: 

• If Idj_| - 0; rank ^ 0' 

• 2if |d^| ^ „ ^] aEffid |d^| ^ |d^ .p. ^1 ; rank = dummy rank 

• IS' |dj_| « |dj_ _ if: cnr Id-j^i = + il or both, count the 
mra&er of d-j^'s thai are eqtial to B, sinn over the associated 
diiaiamy rank£.y then 

. ar^TOl rank:- ^^^^ 

B 

(s3sa jexamples:: A and B on previo-iiES page) 

• Signed "ranks — ass-ocxaire^he sign of the oxtfeginal difference (plus or 
minus) to the actual Trsnks 

• Sum up :irhe positive ranks = T^ 

• Sum up ±he negative isnks = T2 

• DetermiiiB T = minimumiCT^ i ^2) 

•'DetermiTie N = - nuAer of pairs having .a^zzero difference 

N(N + 1) 
Computer yip = ^ 

N(N + 1) (2N 1) 
24 

^ T - pT 
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Output: 

• List of the two input vectors X-^ and X- 

• The sample size for each vector 

• Wilcoxen Signed Rank Test 

• Dat£i table: 







Differ- 




Signed 


Set A 


Set Ti 


ences 


Ranks 


Ranks 


Xll 




di 


^1 




^12 




d2 






• 




• 


• 


• 


• 




• 


• 


• 


• 




• 


• 


• 


^IN 











•'^ Number of pairs (N^) 

- Number of pairs with zesmD difference 

• Test sample size (N) 

• Sum of positive ranks (Tj) 

• Sum negative ranks 

• SmallL sample size test s^tatistic (for cases when ^ 25) 
^ Wilccxxen test statistic (T) 

Larg^ sample size test statistic (N > 25) 
Meau: (y^) 

• Standard ^deviation (drp) 

• Normal a@5JXoximation - Wilcoxen test statistic (Z) 

(Q) Co:chragi:'s Test - to compare K variances for homogeneity. 
-Reference;: Statistics attd. terexliKental Design , Johnson & Leone, Wiley 
19£2, 

^st ithe hjcpoDthesis tfeait K sampfied variaunes are estimates of the 
samtB- populations variant* 



no 



Input : * 

• (Sj^, $2, , - tho sViV'dard deviations lo •^ach of the 

K samples 

• (nj^ , n2, , nj^) - the sample si^es for each of the K 

sampjcs 

Computations: 

• Compare K sample sizes for equality. 

^1 ~ ^^2 " ' * ' " ~ ^ 

• Degrees of freedom = - 1 

• Cochran test statistic 

MAXIMUM 



K 



Output : 

• Summary statistics, sample size and standard deviations for eacii 
of the K samples 

• The Cochran test statistic. 

(8) Least Sio;nificant Difference Test - to compare K means for equality 
Reference:. Experimental Statistics, M.G. Natrella, ^CP 706-110- 
Input : 

• (S]^^, S2^, S^) - the variances for each of tinB K samples 

• (X-j^, X2, Xj^) - the means for each of the JC sai^les 

• (n]^, n2, nj^) - the sampJaEs sizes f or each of the K samples 
Computations : 

fell - 1) + . . . + - X) S k^ 
pooled variance S^^ ^ (ni ^- . . . + uj^ - K 

• haraonic sample size On- = -t^t-t r-7 — ^ 

^ ^ (1/n-^ + - - . + 1/n^ 

K 

• degrees of fjrwwatfbm = I (ri^ - 1) 

i = 1. 

m 



w = maximum difference between any two of the K means 
q = maximum difference converted to units of the standardized range 
statistic 

Output: 

• Summary statistics including S^^-, njj, w, and q. 

• Table of differences between means. 
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AIMS DATA FILES 



113 



TABLE 1 1, la - STUDENT BACKGROUND FILl-: I 



i 



I 
1 



I 

I 



QUANl ITY 


LENGTH 


DIMEN: 


SS No, 


A 


1 


Acad. Id . No . 


2 


1 


*Stud. Name 


16 


1 


Group No . 


1 


1 


Sect. No. 


2 


1 


SAT Math 


2 


1 


SAT Verbal 


2 


1 


CEEB Math 


2 


1 


CEEB Verbal 


2 


1 


Rank 


2 


1 


Sec. Sem. Qpr. 


2 


1 


Thrd. Sem. Qpr. 


2 


1 



One Such Record Per Student. 

*This field is EBCDIC (the only non-binary field in all 8 files). 
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TABLE 11,1b 


- SfUDHlNl BACKGROUND II 




QUANT IIT 


LENGTH 


DIMENSION 


OSPE 


2 


4 


OSPE 


2 


4 


OSPE 


2 


4 


OSPE 


2 


4 


Strong Sc 1" 


2 


65 


Strong Sc 2 


2 


65 


■ 


2 


65 


• 


2 


65 


• 


2 


65 


Strong Sc 65 


2 


65 


EPP 1 


2 


16 


EPP 2 


2 


16 


« 


2 


16 


• 


2 


16 


• 


2 


16 


EPP 16 ' 


2 


16 


Results of Physics 
Validation -Test 


2 





This is a continuation" of the first file. 



One Such Record Per Student. 
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1 

] 

1 

! 

I 



I 
F 

r 



I 
I 
I 
I 
1 
I 

o 

ERIC 



TAHLM TT.2 - STUDKNT PKKFOIIMAMCK I 



QUANTITY L ENGTH ' DIMENSION 

Stud. Id. 2 

Group No. J, 

Sect. .No. 2 

Vol. ■ 1 

No. Ques. 2 

Corr/Inc. 1 20 

Conf. 2 20 

Log Conf. 2 20 

Diff. 2 20 

Tot. Corr. 2 

Avg. Conf. 2 

Avg. Log Conf. 2 

Avg. Diff. 2 



Sort Stud. Id. , Vol. 



One Such Record Per Volume Per Student. 
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I 

I 
I 

1 

I 

I 
I 



r 



I 

o 
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TAB LK TT .3 - STUO K NT PRRFORMANCl': T.I 



QU ANTITY LENGTH ' DIMENSION 

Stud. Id. 2 

Group No. 1 

Sect; .No. 2 

Vol. 1 

No. Ques. 2 

Corr/Inc. 20 

Conf. 2 20 

Log Conf. 2 20 

Diff. 2 20 

Tot. Corr. 2 

Avg. Conf. 2 

Avg. Log Conf. 2 

Avg. Diff. 2 



Sort by Stud. Id., Vol. 



One Such Record Per Volume Per Student 
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I 
I 
I 
I 
I 



1 

I 

I 

I 



I 



TABLE II. A - STUDKNT PERFOl^MAMCK III 
QUANTIl-Y LENGTH DIMENSION 

Stud. Id. ' 2 
Group No, 1 
Sect. No, 2 • 

Corr/Inc. .1 60 



I Tot. Corr. 12 6 

Tot. Corr. 2 2' 6 



Tot. Corr. 3 2 6 

Tot. Corr. 4 2 6 

Tot. Corr. 2 6 



Sort by Stud. Id. 



One Such Record Per Student. 
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I 

I 

1 
I 

1 



1 



er|c , 



TABLE IT.5 ■ MATKRIAL RATING I 

QUANTITY LENGTH DIMENSION 

Vo. ■ 1 ' 

Div. Nos. 2 3 

No Yes 2 

Term Ob j . • 2 20 



\ Item Class 2 20 

Cocc/prin 2 ' 20 

i 

Sort by Vol. , lio. 

I" • • ■ 

One Such Reoarsd Question Per. Volume. 
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TABLE 11.6 - >UTF'UAL RAl ::C "I 



D.i- 

■r:'.rn!n: ; v j . 

.'-'■.Eiirc/pr--r- 
"To Ea; . 
-fcfi'T'g- E-iff . 
ScoT-j ng 1 



-lio. TcmrSirg Cone. 
JSo.. SfeGring TO 



LENGTH 

1 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 



DT^rE^^■s noN 



20 
20 

20 



20 
20 
20 
20 
20 
20 



ScEfe- J>y "Vo 1 . , Div . No . 



One SiiGk. :BfiEord Per Question Per Volume, 



i 
f 
i 
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I 

I 
I 

I 

1 

1 

1 



i^LE 11,7 - MiVIERIAL RAT: ^ III 

QUANTITY LENGTH M MIINSION 

No. Ques. 2 

Term. Ob j . 2 <60 

Ques. Type 2 60 

Sub Id. 2 60 

Corr. Ans. 2 60 

Fac Mth 2 60 

Fac Cone. Z 60 

No Scot* i 60 



Sort by Term, Qlbj 



One Such Record ^«-r ©test ion Pex Volime. 
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I 
I 
I 

i 

1 

1 

1 



QUANTITY 

Stud. Id. 
(Acad. No.) 

Group No . 

Week No. 

Stud. Time 

Mean Time 

Overall Time 



TABLE II. 8 - STITDENTS TIM^ 



LENGTH 

2 
1 
2 
2 
2 
2 



DliMENSlOK 



Sort by Stud. Id., Week, Group No, 



One Such Record Per Volume. 
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3-2 BSER'S DI'SCRIPTI 

te!;NAmiJ:- evaluation: SY^ili 
VOL^ 2 - EESER'S MANUAL 

.1. INTllODUCTION 

This document fntHnde^ tx) provide tfotc ini - rirutioii necessarv 
for the preparalri/on of impiiit to the Aimapol:- s E^^aiuation Sys- 
tem. For a descriptd_on of ithe contents of ;:.ie systteiu as well 
as a description of zrhe -nature oi the tests' Inmvo^ed, consult 
the '^SysteiH Manual — Anirapolis Evaluattion . " 
II. USER INPUT SPECIFICAjEIIM 

A complete tuser input specification consisEts of t\%rc fundamental 
input data blocks. 

Accessing Input - needed to define the: comsttxiiiinGd data 
subset to be used: fmx evaluation. 

Statistical Input — needed to def imie the statistics andf 

or tffiEsts to be ^^erf oraied . 
m: ACCESSING INPUT 

Accessing InpTat consists <oS namixig the siasired ^i^arisble as i3?»^i:i 
as specifying the coiiBStxaiiits which sptw^ to dfeSaaje the (3a£:a 
subset.. Fax: eat^ ruiasstrsiiit a sepaarate r - ^mm t i jmyt^ he submEtdred . 
There is am aZDiro^rle nBaxMnum of 20 dif feirEnt ci^^^pisitxaints :ferr 
each vectox ox mitpnnlt ^<iir»tft^ r 

The first o£ ^arch sett:^^ conditions is thexssKiitfid variable ^mvd. 
each successive card &pfFfnes one constraSftt;* the fcrrmat 1^ 
follows:: 
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File Co.r.inns:: 1 

AC 3-4 

Dimension 1 6-7 

Dimension 2 9 - 10 

Hinimuini 12 lA 

•Jlai^cinsiim 16 18 

l?o3ume " .2.0 -~ 21 



vsihsf^ File, AC, Dinnen?. .xz 1, and Diniensio 2 are described on 
tfe fiollox^ing pages. 

HiinnfTTTiunis and imaximums vrre the lowezrr an~-- urppex bounds desired 
foy tmB tEtsex. The compuo: ir will use thu r::iniirirum. as the lower 
bosumil snd the maximum an the upper bound (i.a, , they will be 
usmi xm the. test, and if one Is hit., the le.t:it Willi be consld- 
eiceiE iSmicciBSsf ul) • Yoliaan^ selection goes rrrnm 3^-115 covering 
v©]lEEimeB A thxcm^h 0 , 

Xf aaa alphabetic charajritiErr is wanted as etrher an upper or 
lower bmimd, l2:s eigrnn^mleiiit number must fee used where A = 1, 
iB = 2-, Z - 2S, ±SL "the 'uiDa. -.me deslgniatiiaiii. 

TDir signify one Bet ni it^ntts is o^sser, the us^ .imiiist put b nine 
iia t&E filfi pomriwa ^esrd J^eave tiaK rest of trfe .card blank, 
m^rm-g: wxne Ti3m.'?=vtL: hs^ btbrfl -■&vai if tlie. user isr^ecLfylng the max±- 
mrnrn mmahexr lof resits (iL^fi. 20) . 

Xbe fell ovmig, fi^ges:. '^vb the File» AC, Dinraarsioa 1, and Dimen- 
s^EBim IL -ftjgr t&agfe piera mS: information. A dash signifies that 
a lfltfafc p?ti'i Li scular f ieHid.^ :^nnot be ussed in a ties t. 
'Jfei^m ILz ISe input caaicfe mian^ be 3ar sequeaoce oir#er, ascendling 
ACGssanfiazng X.o the fil«« t^^te^^, fiJEStt^ fill tilie tdstttts £or file i, 
tf^hsRm SEe 2^ etc.)* fiSQwevea:.., the Etrst c^-irl ;«aust designate the 
•»aiiii(r'*#"rl irarialrlje rega3:.dffiess of whzrt file it: ±s contained! :in (i-. e. 
131525 caxd J^s noil iiiwriiudEd in the aforementirgroed sequence). 
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NOTE 2: For a given rarl iMe constraints can be defined on 
eitthe^r files 1—6 or "—'B (Erati^rial rating files);, but: mot botb. 
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File 


AC 


Dim 


1 


Dim 2 


Fiend 


1 


1 








SS Nuunbcr 


1 


2 








Ac ad Gray Niuimber (Id.) 


1 


— 








Student Name 


1 


4 








(Group KuraScr 


1 


5 








Sectica IJinmbGr 


1 


6 








SAT Math 


1 


7 








;SAT Verbal 


1 


8 








CEEB Math 


1 


9 








CEEB Verbal 


1 


10 








Rank in Class; 


1 


11 








Secondli Seimesiter Qpr 


. 1 


12 








Thirrd SemKssteir Qpr 


1 


-1 


1 






GSPE ill 


1 


-1 


2 






osm #2 


1 


-1 


3 






osm m 


1 


-1 


4 






osm ft 


1 


-2 


1 - 


65, 




strong Scores (1 - 65) 


1 


-3 


1 .- 


16 




Edwards Pearsoninel Preference 
f#*s 1 - 16) 


1 


13 








IPtTFSics Valiidation Test Re- 



COMMEMnr ; 

This record cxccues camce pe'jr stucffipntr smSl is scwrtetf try student Id . 



Field 
Student Id 
Group Number 
Section Number 
Volume 

Number Questions 

Correct /Incorrect 

Confidence 

Log Confidence 

Difficulty 

Total Number Correct 

Average Confidence 

Average Log Confidence 

Average Difficulty 



*This corresponds to the question number since these four fields 
occur for each question • 



Filo 


AC 


Din 


1 


Dim 


2 


1 








2 


2 








2 


3 








2 


4 








2 


5 








2 


-1 


*1 - 


24 


1 


2 


-1 


*1 - 


24 


2 


2 


-1 


*1 - 


24 


3 


2 


-1 


*1 - 


24 


4 


2 


6 








2 


7 








2 


8 








2 


9 









COMMENT ; 

This record occurs once per question and is sorted by Student Id,, 
Volume • 
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Ft 1 p 


AC 


Dim 1 


jj 1 III 


r ic J. CI 


'a 


1 

J. 








■J 


0 






vTl. U 1> LllUU tr L 


3 


•J 








3 


/ 






VnT iiTnp 


3 


5 








3 


-1 


*1 - 24 


1 


Correct / in coT"i'£^ct 


3 


-1 


*1 - 24 


2 


Con "Tt Hpnr p 


3 


-1 


*1 - 24 


3 




3 


_1 


*1 - 24 


4 


Di'f f i ciil tv 

^ Am ^ ^ w y 


3 


6 






Total NuTTiber Correct 


3 


7 






Average Confidence 


3 


8 






Average Log Confidence 


3 


9 






Average Difficulty 



*This corresponds to the question number since these four fields occur 
for each question. 



COMMENT ; 

This record occurs once per question and is sorted by Student Id. , 
Volume . 
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File 


AC 


Dim 1 


Dim 2 


Field 




1 






Student Id. 


U 


2 






Section Number 


4 


-1 


*1 - 64 




Correct /Incon-ect 


4 


-2 


1 




Total Correct Subtest iW 


A 


_2 








4 


-2 


3- 




Total Correct Subtest #3 


4 


-2 


4 




Total Correct Subtest /M 


4 


-2 


5 




Total Correct for Test 



*This corresponds to the question number since this field appears for 
each question. 

COMMENT ; 

This record occurs once per question and is stored by Student Id, 
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File AC Dim 1 Dim 2 Field 

5 1 Voliunc 

5 -1 1-3 Division Nimibers 

5 2 Number of Questions 

5 -2 1 Qu<;^stion Number 

5 --2 2 Form 

5-2 3 Terminal Objective 

5-2 4 Item Classification 

5 -2 5 'Concept/Principle Code 



COMMENT ; 

This record occurs once per question and is sorted by Question #, 
Volume, Form, 



130 



Flic 
6 
6 
6 
6 
6 
6 
6 
6 
6 



AC 
1 
-1 

2 
-2 
-2 
-2 
-2 
-2 

3 

-3 
-A 



Dim 3 
1-3 

1 
2 
3 

5 



*1 _ 7 



*1 - 7 



Dim 2 Field 
Volume 

Division Number 
Number of Questions 
Question Number 
Form 

Terminal Objective 

Item Classification 

Concept/Principle Code 

Number of Faculty Rating 
Difficulty 

Number Scoring Math Category 
■ // 1 - 7 

Number Scoring Concepts 
(Physics) 

Number Scoring Inappropriate 
for TO 



*This corresponds to the category numbers. 



COMMENT ; 

This record occurs once per question and is sorted by Question //, Volume, 
Form. 
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File AC Dim 1 Dim 2 Field 

7 1 Number of Questions 

7 -11 Question Number 

7 —1 2 Terminal Objective 

7 -13 Question Type 

7 ^1 4 Subtest Id. 

7-15 • Correct Answer- 

7 * -2 *1 - 7 Faculty Math Difficulty 

7 -3 *1 - 7 Faculty Math Concepts 

7 2 Number Scoring Inappropriate 

for TO 



*This corresponds to the 7 categories. 
COMMENT : 

This record occurs once per question and is sorted by Question number. 
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I 



I 



File. 
8 
8 
8 
8 
8 
8 



AC 
]. 
2 

■ 3 
4 
5 
6 



Dim 1 



Dim 2 Field 

Student Id. 
Group Number 
Week Number 
Student Tirae 
Mean Time 
Overall Time 



COMMENT ; 

This record occurs once per week and is sorted by Student Id, week. 



ERIC 
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IT.. 2 STATIST IC AI. INPU T 

Statistical input consists of selecting the desired tests and 
computations to be performed on the accessed data as well as 
naming the accessed variables to be used in each test. 

The following contains the detailed instructions for use of 
the statistical package including a list of all required and 
optional data cards, variable definition, and card format. 

HOTE : If the statistical generation routine is called, all 
o£ the single variable computations will be made, except for 
ranking and sorting. If the user desires ranking and sort- 
ing for each vector, he must set ICODEl = 1 in card (4). 
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Card 

No. Rcqun-L-^nd. Cols* 



Ye£: 



Yes^ 



Yes 



1-2 



1-2 



4-5 



Fo mat 



Variable 
Name 



12 



II 



(3N^ 



1-4 
5-8 



12 



12 



12 

F4.0 
F4.0 



LIST 



IR(1) 



IR(2) 



IR(N) 



X(I,J) 



Variab le 
Description 



The number of 
vectors to be 
included in this 
sequence of anal- 
ysis . 

= 1 Print out a 
sunmiary of the 
vectors to be 
analy?.ed and a 
complete set of 
summary statistics , 

= 0 Do not pFxint 
out the above 
summary. 

The number of 
(Observations in 
Vector 1. 

!rhe number of 
cixbservations in 
Vector 2. 



The number of 
observations in 
Vector N. 

The 1^^ data 
print observation 
in the J^^ vector 
where 

J = 1, . . . , N and 
1=1, IR(J), 



NOTE: 



Data cards 2 and. 3 will be included only when inputed from cards. 
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Code VariM)lic Variable 

No. Required Cols. Format Nanc Description 



Yes 1-4 14 IND = 00G3 Compute all 

of the summary sta- 
tistics for each of 
the N vectors being 
analyzed . 

7^ 0003 Do not compute 
the summary statistics 
£o.r each of the N vec- 
3:ors . 



NOTE: If a statistical test requires the nninnpdiai3C:£TOi of summary sta- 
tistics, and IND 4 0003, the control prog^ssL :i5iilEj wt IND = 0003. 



Yes 



5 - 



5-12 



14 



14 



ICOIEI 



IC0DE2 



= ® Do mot compute the 
mc sort each of 

i t^\^ M TPEEttorS. 

- X (Compute tthe rank 

^aesach obaeirvation 
XII eacli vector and 
numesriGally sort each 
vector . 

= 20 Do not compute the 
crostsproducts or corre- 
latanims between any of 
the vectors (called two- 
vector computations) , 



= 21 Compute only the 
crossproducts between 
itdxe vectors. 



13 - 16 



14 



INDICl 



— 122 Cim^aat e the cross- 
Tposoducts 3;nd the cor re- 
IsEtions-iibetween the vec- 

120XS . 

= 0 The two-vector compu- 
tations will be performed 
between all combinations 
of vectors (i.e., to give 
a complete correlation 
matrix) . * ' 



= No. of pairs. If INDICl 
4 0 then it must be set 
equal to the no. of pairs 
of vectors for which the 
two-vector computations 
will be made. 
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Card Variable Variable 

No. Required Cols. Format Name Description 



OPTIONAL 



2-3 
5-6 



(6INDIC1 
- 

(63INDIC1 

(iEMDICl 
-JO - 
(felNDICl); 



12 
12 



12 



IL2 



V(l,l) 
V(l,2) 



VCINDICI, 
1) 



2) 



The first set of vec- 
tors for which Pair 1 
two-vector cotnputatdons 
will be made. 



Pair INDICl, the 
ILasl: set of vector? 
£o r lAich th e t wo-ufi^?/ 
itor cannnpu tat ions TCflLL 
Lbe ussSie . 



.NOTE ; 
6 



Card No. 5 is optional. It is only i^eqiiired if INDICl ^ 0. 



YES 



YES 



1-2 



4-5 



12 



12 



NSTAT 



NVECI 



The code no. for each of 
the statistical tests 
that can be performed per 
the following list: 

40 - t-test 

41 - F-test 

42 - One-way analysis 

of variiance 

43 - Wllfcmgpn Signed 

ltelk:*E3ffi!t 

44 - Keandrng^ Coefficient 

odf GiQBEtcoxdance 

45 - GEd. iSafuare test 

46 - CochxaiEiVs test 

47 ~ Least Significant 

Diffeisence test 

The number^of the first 
vector to be tested. For 
statistical test routines 
40 (t-test) and 41 (F-test). 

The number of vectors to 
be analyzed in each of 
the other test routines. 
For test routines (42-47). 
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Card Variable Variable 

•No. Required Cols. FbmiaL Name DcscripLioa 



ItES 7-8 12 NVEC2 The number of the 

second vector to be 
tested for statistical 
routines O\0 & Al) . 

This field is blank 
for all of the other 
statistical test rou- 
tines (42-47). 

'OPTMNAL li - 80 20A4 NAME User supplied des- 

cription of variables 
(vectors ) be i.ng tes ted 
within a particular 
test routine 

MOTS: The descripition is optional; a blank card must be substituted if 
user decides not to use the description option. 



8 



TES 



3-4 

7-8 



(4NVECX 
-1) - 
(4NVECL) 



12 

12. 



NN(1) 
NN(2) 



NN(NVEC10 



The number of each of 
the vectors to be in- 
cluded in the analysis 
for each of the statis- 
tical test routines. 



NQIlE:: if statistical tesiti routines 40 or 41 are specified, do not in- 
cJanfe- control card 8 or 9.. 

WEE: Control card 9 is only required for statistical test routine 45 
(Chi Square test) . This card provides the user with option of defining 
the cell size for the Chi Square test or the option of selecting the 
nuniber of ceMs to be u3ed in ansSLysis. 



YES 



m 



LL 



- 1 If the user desires 
to define the limits 
for each cell (define 
the cell size) . 



= 2 If the user does 
not define the limits 
for each cell. 
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Card 
. No. 



Cols. 



Formvit 



Vaclablc 



Variable 
DGscrlption 



3-4 



5 - 10 
11 - 16 



(12Kr-2:) - 
(lZK+4) 



12 



F6.0 
F6.0 



F6.0 



ser limit for the 
cell (interval) , 
if IZXK.32) = 99999. 
Tte oipper limit for 
tha^TlEth interval is 
set: iro +00. 



JNOTE.: On Chi Square card, xf the user s:elects EL = 2 and sets K = 0, 
then the program vill assume six cells for the Chi Sqiuare analysis by 
selecting the minimum (MIN) and maximum (MAX) observations from all of 
the observations available (under analysis) and define the cell width. 



LI (1,1) 
LI(l32) 



LI (K,2) 



The number of cells 
to be used in the 
Chi Square analysis. 
If LL ~ 1, R must be 
specified- If LL = 2, 
K need not be speci- 
fied (sec Chi Square 
note at ead of this 
table . 

Lower limit for the 
first cell if LI 
(1,1) = 99999. The 
lo wer limit for the 
first cell = -00 



A 



MAX - MIN 



and define each of the six cells as follows: 



Ce 
Lower 
Limit 


11 1 

Upper ' . 
Limit 


. Ce3 
Lower 
Limi t 


la 2 

Upper 
Limit 


• • • 


Cei: 
Lower 
Limit 


L 6 
Upper 
Limit 


MIN 


MIN-fA 


■ MIN+A 


MIN+2A 




MINfSA 
MAX-A 


MAX 
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If the user selects LL = 2 and specifies K (the nur.iber of cells) as some 
integer T then the cell width A will be defined in a like manner. 

MAX - MIN 



and each of the i cells will be defiinod as follows: 



Cel 

Lower 
Limit 


1 1 

Upper 
Limit 


• • • 


Cel 

Lower 
Limit ; 


1 1 

Upper 
Limit 


. MIN 


MIN4-A 






MAX 



Example Set of Control Cards 

Test case given 10 vectors of da>ta (J = 1, ...» 10) each of size IR(J) : 

1. Compute summary statistics; ^aod M;5t. 

2. Compute correlation for vectxso: sets (1 and 2^^^^ O snd 4) 3 (5 and 6) , 
(7 and 8) , (9 and 10) . 

3. Perform t-tes.t om vectors 2 sad 4. 

4. Perform Kendall Coefficient of Grancordance on vectors 1, 3 3 5, 
and 9. 

5. Perform Chi Square [with 8 unspecified iTitervaiLs (^Efil l s) ] for 
all 10 vectors. 

6. Chi Square tesi: C^same vectors witk 3 specifxed mterv^als (0,10) 
(10,40) 440,00). 

A coding sireet is attached with the IS control caosds necresrsary to perform 
the above analysis . 

NOTE: Control (data) cards 2 and 3 far 1:he number of ofeervations in each 
vector have not been included. 
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3 . 3 SUPPLKMK NT 

ANNAPOLIS EVALUATION SYSTEM 
VOL. A - SUPPLEMENTARY WORK - ITEM ANALYSIS, PMPG 

I . INTRODUCTION 

In the course Of the Annapolis experiment > programs were generated 
to perform specific evaluative tasks, not included in the main- 
line system (Annapolis Evaluation, Vols. 1, 2> and 3). It is 
the purpose of this document to describe the nature and use of 
these programs, the Item Analysis Program, and PMPG (Percentage 
Maximum Possible Gain) Program. 
II. ITEM ANALYSIS 
II . 1 Program Description 

The program performs the following analyses for student response 
data: a) for all students or keyed groups of students, and b) 
on all of the test questions or any keyed subset of questions. 
A. Summary Statistics: 

1. Xounts the number of students taking the- test. 

2. Computes- the raw score (no. of items correct) 
for each student. The raw score is computed 
based on the number of keyed items. 

3. Computes the average raw score for all stu- 
dents (test average). 

A. Computes the standard deviation of student 
scores. 
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B. Item Sti.itistics (for each keyed item): 

1. Conpules the frequency and perccnlage oi: the 
upper 50% of students' responses for each 
category of response. 

Categ;orles of Response 

Fill-in questions X - Correct 

0 - Incorrect 

M - Blank 
Multiple-choice questions 

B I Standard multiple 
C ^ choice categories 
D 
E 

M - Blank 

2. Computes the frequency and percentage of lower 
50% students for each category of response, 

3. Computes the frequency and percentage over all 
students for each category of response. 

4. Computes the difference in percentage between 
upper and lower students for each category of 
response . 

5. Computes the average rax^ score for students who 
answered the question correctly (AC) . 

6. Computes the average raw score for students who 
answered the question incorrectly (AIC) ' 

7. Computes the proportion of students who answered 
the question correctly (PPS) . 
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8. Computes Lhe item (quesLlon) sLandard deviation 
S - [(PPS) (] .0 - PPS)]^/^' 

9. Compute the point biserlal correlation 

PBC ^ (AC - ATC) : — — ; : — 

lest stanaard & deviation 

Additional computations and processing: 

1. Computes the Kuder-Richai:dson reliability (KR20) 

i^p9n N . (SS)^' - SSI 
N - 1.0 (SS)2 

N = number of keyed items. 

Where : SS = test standard deviation 

SSI = S (PPS) (1.0 - PPS) 

2. Sorts the proportion passing for each keyed Item 
into ascending order. 

3. Sorts the point biserial correlation for each 
keyed item into ascending order. 

4. Sorts all participating students into descending 
order by grade. 

II. 2 Program Use 

The item analysis program was developed specifically for anal- 
ysis of the Final Examination. The program via user control 
cards can perform an item analysis on any subset of keyed 
questions for all students or for groups of students. The 
program was written in Fortran !IV for use on the IBM 360/65 
under OS. The control and data cards for analysis of the 
Final Exam are as follov/s : 
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Card No. 



Co 1 . No . 



Description 



(one card 
for each 
student) 



1-60 

1-2 
3 

4 

62 

1-4 

7-10 
11-12 

13 
14 



Literal di-^scr ipt ion of the test to 
he ana lyxed . 

Number of questions on the test. 

The correct responses , one for 
each question on the test. The 
follov\?ing are acceptabli^. data 
values for each question: Fill- 
in question: X - s>niibolica lly 
for the correct answer. . 
Multiple-choice question: A, B, 
C, D, or E - vjhichcver is the 
correct response . 

The section number of the students. 

The student's identification no. 

The raw score for the student (no. 
of questions answered correctly. 

Student's response to each question 
corresponding to the following cate- 
gories : 

Fill-in questions: X - correct re- 
sponse, 0 - incorrect response, M - 
no response. 

Mult iple- choice questions : 

A, B, C, D, or E - whichever the 

student responded, M - no response. 



NOTE: After the last student card, a card with zero's in Columns 7-10 
is required to indicate that the last student has been processed. 



1-2 
5-6 



The number of questions to be analyzed 

The number of class sections to be 
included in the analysis. 



NOTE: If all students are to be included in the analysis columns, 5-6 
should be left blank. 



1-2 
3-4 



The first question to be analyzed. 
The second question to be analyzed, 



79-80 
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NOTE: If all the qm^stion on ihc test nre to be nnnlyxcd, card 5 is not 
required . 

Card No. Col. No. Description 

6 1-^ The first class section to be inclu- 

ded in analysis . 

5-8 Tlie second class section to be inclu- 

ded in this analysis 



77-80 

The card sequence (4^ 5^ 6) can be repeated as often as required by the 
user . 

III. PERCENTAGE MAXIMUM POSSIBLE GAIN 

111.1 Introduction 

A computer program to calculate the percentage of maximum possi- ' 
ble gain (PMPG) was developed per the specifications contained 
in a memo by Robert L; Brennan, entitled "Statistical Techniques 
for Measuring Effectiveness and Gain with Instructional Programs." 

111. 2 Program Description 

The PMPG program computes the following statistics for student 
post- and pretest data for each week (or course unit) 
A. Summary Statistics: 

1. The number of questions on the test 

2. The number of students taking each test 

3. A frequency histogram of student scores 
for each test 

4. The computed PMPG 
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B. PMPG Co::;putation 

As specified in the above referenced document, the 

percentage of maximum possible gain is computed from 

the following equation: 

Ny y - 1 
Ny Z 2 *y (i) 

PMPG = 1 - ^ X - 1 ^ ° - ^^^^^ - ^ 

k '2 (i) 

X = 1 1 = 0 

where: Nx = no, of questions on the pretest 

X (i) = ? X = i = probability that a 
student scores i questions right on the 
pretest 

Ny = no. of questions on the posttest 

y (i) = P Y = i = probability that a 
student scores i questions right on the 
posttest. 



III. 3 Prof^ram Use 



A- The PMPG program* requires two magnetic tape files, one 
for pretest data and the other for posttest data. The 
program was written for the specific case of the pre- 
and posttest magnetic tape data files that were created 
for the Annapolis evaluation. The student records were 
sorted on tape alphabetically by volume (for volumes A 
through 0). In addition, a test value of -1 separated 
each volume (the student records for each volume from 
the student records for the next volume). 

B. The program was written in Fortran IV for use on the IBM 
360/65 under DOS. 
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C. The following are the input card and tape foritints used: 
1, Card Format 



Card No. Col. No, Description 

1 1 Vol, = The volume to be 

processed 

3-5 NPRE = no. of questions 

on the pretest 

6-8 NPOST = no. of questions 

on the posttest 

2, Tape record format (for both post- and pretest 

tapes) 

Record Format Description 

A2 The end of volume test value 

(= -1 indicates the last re- 
• cord for that volume has been 
processed 

3X 

Al The volvnne letter of the cur- 

rent record 

140X 

A2 The student's score (no, of 

\ questions answered correctly 

for this volume. 
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3.4 COPIES OF PROGRAMS 



Copies of the prograras were submitted by 
Gontractural requirement on 1) a magnetic 
tape and 2) hard copy listings. Copies 
of the p^rograiii^v are referenced as Anna- 
polis Evaluation, Vol, 3, 
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